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CHAPTER  I 

INTRODUCTION 

Rationale 

The  Importance  of  Reading 

Of  all  the  skills  one  acquires  as  a result  of  formal 
education,  the  ability  to  read  must  certainly  be  Ranked 
as  one  of  the  most  important.  Literacy,  which  is  broadly 
defined  by  Bormuth  (1973-1974)  as  nothing  more  than  the 
ability  to  respond  appropriately  to  written  language  (the 
ability  to  read)  always  has  been  given  prominent  emphasis 
by  various  social  movements  seeking  to  improve  the  status 
of  mankind.  Reading  has  received  such  emphasis  because 
of  an  acute  awareness  that  little  in  life  of  any  great 
value  is  ever  accomplished  without  in  some  way  involving 
reading . 

Because  of  its  great  importance,  the  active  develop- 
ment of  reading  is  begun  in  children  shortly  after  they 
enter  elementary  school.  Initially,  students  read  prima- 
rily for  practice.  Very  soon,  however,  as  more  and  more 
knowledge  is  transmitted  through  the  medium  of  written 
language,  academic  survival  becomes  primarily  dependent 
upon  the  ability  to  read. 
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The  Importance  of  Foreign  Language  Reading 

As  in  all  other  academic  disciplines,  reading  also 
plays  a central  role  in  the  learning  of  a foreign  language. 
Although  oral  skills  frequently  are  emphasized  initially 
in  many  foreign  language  programs  today,  primarily  as  a 
result  of  the  influence  of  the  audio- lingual  method,  which 
gained  so  much  popularity  in  the  1960s,  Phillips  (1974) 
points  out  that  nevertheless,  relatively  quickly  students 
begin  to  devote  considerable  time  to  reading.  For  example, 
even  while  speaking  skills  are  being  given  the  dominant 
emphasis,  speaking  practice  in  class  often  centers  around 
assigned  reading  passages.  Beyond  the  beginning  level, 
of  course,  various  reading  activities  receive  a major  portion 
of  the  student's  attention.  So  important  is  reading  to 
the  overall  foreign  language  classroom  environment,  that 
Phillips  (1974)  indicates  that  its  elimination  "would  make 
it  extremely  difficult  to  teach  any  language  skill  in  the 
traditional  school  setting"  (p.  1).  Ultimately,  then,  being 
a successful  second- language  reader  most  likely  determines 
student  achievement,  in  the  total  foreign  language  program, 
more  than  any  other  skill. 

Because  of  the  centrality  of  reading,  it  would  there- 
fore seem  most  apparent  that  any  pedagogical  practice  or 
procedure  designed  to  assist  in  either  refining,  improving, 
or  evaluating  student  reading  skills  would  be  welcomed  by 
both  teachers  and  students  alike. 
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Problem  Statement 
Reading  and  Comprehension 

For  years  educators  have  struggled  with  the  problem 
of  defining  what  skills  are  involved  in  reading.  Over 
30  years  ago,  Davis  (1944)  pointed  out  that  there  was 
no  widespread  agreement  as  to  exactly  what  reading  abil- 
ity was.  Today,  there  is  still  no  widespread  agreement. 
Authorities  generally  agree  only  that  reading  consists 
of  a collection  of  different  skills.  Some,  however,  have 
postulated  large  numbers  of  skills,  as  for  example  Burkhart 
(1945),  who  identified  214  separate  skills;  while  others 
have  preferred  to  define  reading  in  terms  of  a relatively 
small  number  of  skills.  Anderson  (1949),  for  example, 
using  statistical  techniques,  limited  reading  to  just  three 
factors . 

Although  educators  and  reading  researchers  over  the 
years  have  been  unable  to  agree  upon  exactly  which  skills 
are  involved  in  reading,  there  nevertheless  has  been  general 
agreement  that  the  student  must  be  able  to  understand  what 
he  or  she  reads.  Bormuth  (1969)  indicates  that  the  ability 
to  comprehend  is  one  of  the  most  basic  factors  determining 
the  effectiveness  of  instruction.  He  says  that  if  the 
student  is  unable  to  understand  the  language  of  the  school's 
curriculum,  "he  will  almost  certainly  fail  to  learn  much 
of  the  content  of  his  instruction  and  both  the  student 
and  the  instruction  will  fail  to  attain  their  objective" 

(p.  3). 
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In  discussing  foreign  language  reading,  Allen  and 
Valette  (1972)  suggest  that  reading  involves  more  than 
just  assigning  foreign  language  sounds  to  the  written 
words.  Instead,  "it  requires  the  comprehension  of  what 
is  written"  (p.189).  Chastain  (1971)  points  out  that 
the  ultimate  goal  in  reading  a foreign  language  is  to  be 
able  to  read  comfortably: 

The  word  "comfortably"  implies  that  the  stu- 
dents should  reach  a level  at  which  they  do  not 
feel  a conscious  strain  while  reading,  i.e.,  a level 
at  which  they  can  concentrate  on  the  message  with- 
out being  consciously  aware  of  the  code.  Naturally, 
students  who  spend  most  of  their  time  thumbing 
through  a dictionary  while  preparing  an  assign- 
ment have  not  obtained  a reading  level  consistent 
with  the  aforementioned  objective  (p.  177). 

Although  there  is  no  widespread  agreement  as  to 
exactly  what  skills  are  involved  in  reading,  it  is  never- 
theless evident  that  the  student  must  be  able  to  understand 
the  reading  material.  In  fact,  in  foreign  language  in- 
struction, one  cf  the  key  criteria  in  selecting  reading 
passages  ultimately  must  be  whether  or  not  the  material 
can  be  comprehended.  Because  the  reading  skill  is  so 
central  to  successful  foreign  language  learning,  great 
care  must  be  exercised  by  the  teacher  in  choosing  reading 
selections  that  are  comprehensible. 

Readability 

One  key  factor  that  is  often  underemphasized  in  de- 
termining suitable  reading  selections  is  the  difficulty 
level  of  the  passage,  i.e.,  its  readability.  Unfortunately, 


foreign  language  students  are  frequently  introduced  to 


reading  selections  far  too  advanced  for  their  reading  capa- 
bilities. At  the  very  least,  such  situations  result  in 
extreme  frustration,  which  if  unchecked,  may  lead  to  dis- 
interest and  possible  disenrol lment- -a  tragic  occurrence 
under  any  circumstances. 

In  examining  the  phenomenon  of  reading  difficulty, 
one  quickly  recognizes  that  it  is  a most  complex  problem 
influenced  by  a wide  variety  of  factors.  Anderson  (1976) 
suggests  that  readability  factors  can  be  divided  broadly 
into  two  categories:  material  factors  and  subject  factors. 
For  example,  reading  difficulty  is  undoubtedly  influenced 
by  such  factors  as  vocabulary,  the  complexity  of  the 
sentence  structures,  the  amount  of  redundancy  present, 
the  style  employed  by  the  writer,  the  format  utilized, 
typographical  considerations,  etc.,  in  other  words,  all 
those  factors  that  are  inherent  in  the  material  itself. 
Additionally  significant,  however,  are  those  subject  factors 
related  directly  to  the  reader  himself  such  as  intelligence, 
aptitude,  attained  reading  level,  motivation,  experience, 
etc.  That  both  of  these  categories  of  factors  are  very 
important  in  determining  reading  difficulty  levels  has 
been  a well  known  fact  for  some  time.  Almost  thirty  years 
ago,  Betts  (1949)  indicated: 

Reduced  to  its  lowest  terms,  readability  is 
a two-way  proposition.  The  first  consideration 
is  the  reader--his  experience,  his  interests,  his 
feelings,  his  motivation,  his  language  facility, 
his  needs,  and  Iris  reading  and  study  conditions. 

Any  means  of  predicting  readability  is  valid  to 
the  degree  that  the  reader  is  taken  into  account. 
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The  second  consideration  is  the  interest  level, 
the  language,  the  mental  constructs,  and  the  mech- 
anical features  of  the  reading  material  (p.  438). 

Most  recently,  the  importance  of  the  interaction 

between  reader  and  materials  has  been  stressed  by  Anderson 

(1976),  who  indicates  that  "any  measure  of  readability 

should  take  these  two  factors  (material  and  subject)  into 

account"  (p.  5). 

Considering  the  complexity  of  accurately  assessing 
readability,  which  clearly  involves  both  material  factors 
and  subject  factors,  a most  significant  question  is  raised: 
How  can  teachers,  especially  foreign  language  teachers, 
know  if  reading  selections  they  have  chosen  are  of  an  ap- 
propriate difficulty  level  for  their  students? 

English  Readability  Studies 

The  topic  of  assessing  reading  difficulty  has  been 
an  area  of  interest  and  concern  for  many  years.  According 
to  Lorge  (1941),  the  first  recorded  attempts  to  apply 
readability  procedures  were  made  by  the  Talmudists,  who, 
as  early  as  900  A.D.,  made  word  and  idea  counts  to  dis- 
tinguish usual  from  unusual  meanings.  In  addition  to 
religious  teachers,  Gray  (1937)  indicates  that  educators 
began  studying  elements  of  reading  difficulty  centuries 
ago  in  connection  with  children's  reading.  Evidence  of 
this  interest  appeared  as  early  as  1840  when  ease-of-under- 
standing  in  the  McGuffey  Readers  was  considered  in  terms 
of  vocabulary. 
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More  modern  studies  of  English  readability  date  back 
to  the  early  1920s.  Chall  (1958)  awards  credit  for  the 
first  quantitative  study  in  readability  to  Pressey  and 
Lively,  who  published  a paper  on  vocabulary  burden  in 
1923.  Since  that  time,  the  problem  of  readability  has 
received  ever-increasing  attention  from  educational  re- 
searchers . 

Most  widely  pursued  has  been  the  concept  of  the 
readability  formula.  Defined  by  Klare  (1963)  as  "a  method 
of  measurement  intended  as  a predictive  device  that  will 
provide  quantitative,  objective  estimates  of  the  style 
difficulty  of  writings"  (p.  3),  the  aim  of  the  readability 
formula  has  been  viewed  as  a fairly  accurate  yardstick 
for  measuring  the  reading  difficulty  of  a given  text. 

Some  of  the  more  prominent  early  researchers  who  have 
developed  popular  English  readability  formulas  include: 

Lorge  (1939),  Flesch  (1943  and  1948),  and  Dale  and  Chall 
(1948) . 

Foreign  Language  Readability  Studies 

Regarding  foreign  language  readability  studies,  Klare 
(1963)  indicates  that  most  research  has  been  done  in  the 
United  States.  The  earliest  work  detected  by  this  researcher 
was  a study  conducted  in  1927  by  Johnson,  who  suggested 
that  French  could  be  rated  according  to  difficulty  by 
determining  the  number  of  "uncommon"  words  per  1,000  running 
words.  In  reviewing  foreign  language  readability  research 
up  to  1950,  Spaulding  (1950)  was  able  to  uncover  only  a 


handful  of  studies,  all  of  which  were  conducted  in  either 
French  or  Spanish. 

Concerning  the  development  of  actual  foreign  language 
readability  formulas,  only  three  were  uncovered:  1)  Ac- 
cording to  Hall  (1971),  the  first  readability  formula 
devised  specifically  for  foreign  language  materials  was 
developed  in  1939  by  Tharp,  who,  by  utilizing  the  factors 
of  density  and  frequency,  tried  to  determine  the  relative 
difficulty  of  French  texts  for  English-speaking  readers. 

2)  The  most  widely  known  foreign  language  readability 
formula  was  published  in  1956  by  Spaulding,  who  simplified 
a formula  he  had  devised  in  1951  for  use  in  Spanish. 

Based  on  the  Dale-Chall  formula  (1948),  Spaulding’s 
formula  contained  two  variables:  average  sentence  length 
and  vocabulary  density.  3)  In  1963  Scherer  modified 
and  adopted  Spaulding's  formula  for  use  on  German  texts 
intended  for  English-speaking  readers. 

Readability  Formula  Weaknesses 

Although  the  idea  of  assessing  readability  has  been 
a topic  of  concern  for  centuries,  only  during  the  last 
50  years  has  the  subject  been  seriously  pursued.  While 
only  limited  findings  have  been  recorded  involving  foreign 
language  reading  difficulties,  the  bulk  of  the  readability 
research  has  been  accomplished  in  English  and  has  centered 
around  the  development  of  the  readability  formula.  Despite 
the  fact  that  prominent  researchers  have  developed  popular 
formulas,  however,  difficulties  have  remained.  In  discussing 
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the  problems  of  readability  formulas,  Bormuth  (1967a) 
points  out  the  following: 

Perhaps  one  of  the  chief  reasons  why  instruc- 
tional materials  are  not  routinely  evaluated  to 
determine  whether  they  have  a suitable  level  of 
difficulty  is  that  there  has  been  no  technique  that 
is  at  once  convenient,  economical,  and  valid. 
Readability  formulas  are  convenient,  inexpensive, 
and  require  only  unskilled  clerical  assistance  to 
use,  but  the  formulas  presently  available  have 
validities  that  range  from  .5  to  only  about  .7. 
Moreover,  the  equations  take  into  account  only  a 
limited  range  of  linguistic  variables  and  the 
variables  that  are  taken  into  account  are,  by 
today's  standards,  crude  (pp.  2-3). 

Why  have  readability  formulas  generally  experienced 
validity  difficulties?  Two  reasons  are  immediately  ap- 
parent: First,  as  Bormuth  suggests,  only  a limited  range 

of  linguistic  variables  has  been  considered,  and  these 
are  rather  crude  by  today's  standards.  The  second  reason 
is  found  in  something  Bormuth's  statement  implies.  That 
using  relatively  crude  linguistic  variables  to  assess 
reading  difficulty  levels  would  decrease  the  valitity  of 
a readability  formula  is  readily  understandable.  More 
important,  however,  is  the  fact  that  only  linguistic 
variables  have  been  considered.  As  Anderson  (1976)  has 


indicated,  the  area  of  readability  must  be  divided  into 
both  material  factors  and  subject  factors.  Because 
traditional  readability  formulas  have  focused  primarily 
on  material  factors  and  have  almost  totally  neglected 
subject  factors,  it  is  little  wonder  that  validity  levels 
have  ranged  no  higher  than  between  .5  and  .7.  If  in 
assessing  reading  difficulty,  validity  levels  of  readability 
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formulas  are  to  be  improved,  additional  factors  must  be 
taken  into  consideration. 

Direct  Testing,  Another  Possibility 

If  traditional  readability  formulas  arc  incapable 
of  accurately  and  validly  assessing  reading  difficulty 
because  their  linguistic  variables  are  crude,  and  because 
they  fail  to  take  into  account  subject  factors,  what  other 
options  are  available?  According  to  Bormuth  (1975) , 

"the  time  honored  way  to  find  out  if  a selection  is 
suitable  for  students  is  to  test  them  on  it"  (p.  62). 

To  do  this,  the  teacher  identifies  the  appropriate  reading 
selection,  writes  some  questions  over  the  material,  has 
the  students  read  the  material  and  then  try  to  answer 
the  questions,  and  then  scores  the  test.  Finally,  some 
criterion  score,  say  75  percent,  is  used  to  interpret 
student  performance  by  supposedly  representing  a mini- 
mum level  of  desirable  performance.  For  those  who  score 
at  or  above  the  criterion  score,  the  material  is  con- 
sidered readable  and  therefore  suitable. 

Direct-Testing  Weaknesses 

It  is  evident  that  direct  reading  tests  take  into 
account  both  material  factors,  because  the  passage  of  concern 
is  actually  read,  and  subject  factors,  because  it  is  the 
students  themselves  who  are  doing  the  reading.  Nevertheless, 
several  problems  arise  from  following  such  a procedure. 

First,  there  is  strong  evidence  that  tests  which  teachers 
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write  are  not  reliable  measures  of  content  difficulty  alone. 
Long  ago,  Lorge  (1949)  criticized  direct  testing  because 
it  was  uncertain  whether  the  actual  difficulty  level  of  the 
passage  was  reflected  in  the  difficulty  of  the  language 
used  in  the  test  items  themselves.  For  example,  a diffi- 
cult selection  may  appear  easier  if  the  test  questions 
the  student  must  answer  are  simply  phrased.  Similarly, 
an  easy  passage  may  appear  difficult  if  the  test  questions 
are  couched  in  more  difficult  language  than  the  language 
of  the  passage  itself.  Such  factors,  of  course,  are  deter- 
mined by  individual  teachers.  Bormuth  (1975)  says: 

Test  writers  influence  the  difficulty  of 
tests:  two  writers  making  a test  over  a single 

passage  could  produce  tests  of  quite  different 
difficulty,  the  one  writer's  test  eliciting  mostly 
low  scores  and  the  other's  mostly  high  scores  (p.  63). 

Although  directly  testing  students  on  passages  they 
have  read  seems  more  valid  than  readability  formulas  as 
a measure  of  reading  difficulty,  such  tests  are  never- 
theless unreliable  and  suspect  because  the  test  items 
themselves  also  represent  a reading  task,  the  difficulty 
of  which  is  determined  by  individual  teachers  and  not 
by  the  reading  selection  itself.  Because  of  this,  Bormuth 
(1967b)  indicates  that  "it  is  uncertain  whether  it  is 
the  difficulty  of  the  passage  or  the  difficulty  of  the 
(test)  items  that  is  measured  by  this  procedure"  (p.  5). 

A second,  more  subtle,  problem  associated  with 
directly  testing  the  student's  comprehension  of  a reading 
selection  involves  the  manner  in  which  the  test  items  are 
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selected  for  inclusion  in  the  test.  Bormuth  (1970)  states 
the  following: 

The  traditional  item  writer  seldom,  if  ever, 
sets  out  to  write  every  possible  item  of  a given 
type  in  making  a test  over  a program.  Rather,  he 
avoids  writing  items  he  considers  trivial , too 
complex,  too  simple,  too  wordy,  or  otherwise  un- 
desirable. Similarly,  he  will  often  simply  decide 
that  he  has  produced  as  many  items  as  he  needs.  His 
decisions  on  these  matters  are  seldom  explicit  and 
are  almost  never  rationalized.  Thus,  as  the  test 
writer  generates  items  by  traditional  methods,  he 
is  implicitly  designing  the  test,  but  doing  so  in 
a manner  that  is  not  open  to  inspection  and  rational 
review  (p.  13).  ingun 

Because  any  set  of  questions  chosen  to  test  how  well 
a student  has  comprehended  a reading  selection  is  nothing 
more  than  a sample  of  all  the  possible  test  questions 
that  could  be  written  over  the  selection,  it  is  extremely 
important  that  the  sample  of  test  questions  be  an  unbiased 
one.  Unfortunately,  however,  because  the  manner  in  which 
questions  are  selected  is  so  unrationalized  and  non- 
operational,  there  is  no  way  to  be  certain  that  the  final 
test  is  either  biased  or  unbiased.  There  therefore  arises 
another  source  of  uncontrollable  error  variance  that  can 
have  marked  effects  on  the  outcome  of  the  test. 

A third  problem  in  directly  testing  comprehension 
involves  the  selection  of  acceptable  criterion  scores. 
According  to  Bormuth  (1969),  reputable  reading  authori- 
ties such  as  Betts  (1946),  Bond  and  Tinker  (1967),  and 
Harris  (1962)  claim  that  reading  materials  are  suitable 
for  unsupervised  study  if  a student  can  score  at  least 
90  percent  on  a test  over  their  content.  Materials  arc 
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regarded  as  suitable  for  supervised  study  if  the  student 
can  score  75  percent.  Bormuth  further  indicates  that 
a search  of  the  literature  has  revealed  that  those  two 
criterion  scores  (75  percent  and  90  percent)  originally 
could  be  traced  back  to  Thorndike  (1917),  who  apparently 
acquired  them  from  teachers  who  had  begun  using  them  as 
a result  of  oral  tradition.  Interestingly,  however,  is 
Bormuth' s (1975)  observation  that  none  of  these  reading 
experts  has  ever  offered  any  evidence,  reasoning,  or 
rational  support  for  selecting  these  particular  scores. 

In  fact,  Bormuth  (1967b)  states  that  traditionally  the 
selection  of  a criterion  score  has  been  viewed  "as  a 
matter  to  be  left  to  personal  preference  or  arbitrary 
choice  rather  than  as  a matter  for  rational  decision 
based,  at  least  in  part,  on  empirical  data"  (p.  22). 

Bormuth  (1975)  further  suggests: 

We  are  left  to  wonder  if  the  criterion  scores 
sprang  from  an  arbitrary  decision  that  became 
standard  practice  simply  because  of  the  strength 
of  precedence.  The  length  of  the  English  unit  of 
measure,  the  foot,  took  on  its  value  because  the 
king  who  standardized  that  measure  happened  to  have 
a foot  of  that  length.  The  chief  effect  of  this 
accident  of  history  was  to  saddle  us  with  an  amazingly 
inconvenient  measurement  system  (p.  64). 

From  all  of  the  foregoing,  it  should  be  readily  ap- 
parent that  there  is  no  empirical  evidence  of  any  sort 
suggesting  that  a criterion  score  of  either  75  percent 
or  90  percent  makes  a passage  ideal  or,  for  that  matter, 
even  acceptable  for  instructional  use. 
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The  Cloze  Procedure,  A Third  Possibility 

Recognizing  that  both  readability  formulas  and  direct 
tests  of  reading  comprehension  possess  inherent  weaknesses 
that  can  bias  and  therefore  invalidate  findings,  where 
next  must  one  look  in  seeking  a way  to  objectively  assess 
the  difficulty  level  of  reading  selections? 

A procedure  that  has  demonstrated  great  promise  for 
assessing  the  difficulty  of  both  native  and  foreign  lan- 
guage reading  selections  is  the  cloze  test.  Introduced 
by  Taylor  (1953),  who  adopted  the  word  "cloze"  from  the 
Gestalt  psychologists'  concept  of  closure,  the  cloze  pro- 
cedure is  a way  of  constructing  tests  by  mechanically  and 
systematically  deleting  the  words  in  a reading  selection 
and  replacing  them  with  an  underlined  blank  of  standard 
length.  Students  being  administered  the  tests  are  then 
expected  to  guess  what  word  has  been  taken  out  of  each 
blank  and  write  it  in  that  space.  There  is  a variety  of 
ways  to  select  the  words  to  be  deleted.  According  to 
Bormuth  (1973-1974),  "what  distinguishes  a cloze  test  from 
an  ordinary  deletion  test,  though,  is  the  fact  that  in 
a cloze  test  the  words  may  be  selected  for  deletion  only 
by  a completely  replicable  set  of  rules"  (p.  38). 

Upon  close  examination,  one  fact  becomes  immedi- 
ately apparent  regarding  the  cloze  test:  Because  it  is 
a technique  that  measures  how  well  n reader  has  under- 
stood what  he  has  read,  it  may  he  used  both  as  an  index 
of  a passage's  readability,  i.c.. 


its  reading  difficulty. 
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and  as  an  index  of  a student's  ability  to  comprehend  the 
passage.  This  characteristic  is  important  because  it 
indicates  that  both  material  and  subject  factors  are  taken 
into  consideration,  which,  of  course,  is  essential  in 
order  for  any  reading  measurement  test  to  be  valid  and 
reliable.  According  to  Anderson  (1976): 

The  significance  of  cloze  procedure  as  a 
measure  of  readability  is  that  the  method  appears 
to  take  into  account  both  material  and  subject 
factors  of  readability.  There  is  a sound  theo- 
retical basis  also  for  cloze  procedure  as  a measure 
of  reading  comprehension  for  cloze  scores  index 
the  correspondence  of  language  habits  of  writer 
to  those  of  reader  and  thus  the  construct,  reading 
comprehension,  is  given  operational  meaning  (p.  7). 

The  cloze  test  appears  to  have  the  necessary  quali- 
fications for  adequately  dealing  with  both  material  and 
subject  factcrs  in  assessing  reading  difficulty.  In 
addition,  however,  unlike  direct  tests,  which  are  tra- 
ditionally influenced  by  the  various  biases  of  the  indi- 
vidual test  writer  who,  in  generating  questions,  can 
make  the  reading  selection  appear  either  more  difficult 
or  easier  than  it  actually  is  and  thereby  reduce  the 
validity  of  the  findings,  cloze  tests  do  not  inject 
irrelevant  sources  of  variance  into  the  process  because 
the  manner  in  which  the  cloze  test  is  generated  is 
operationally  defined  in  advance.  Therefore,  the  test 
reflects  only  the  actual  difficulty  level  of  the  reading 
selection  under  consideration.  For  obvious  reasons,  such 
a procedure  yields  far  more  valid  results  than  would  be 
had  by  using  either  a traditional  readability  formula 
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or  a direct  test  of  reading  comprehension. 

A Problem  with  the  Cloze  Test 

Despite  the  fact  that  the  cloze  procedure  deals  with 
both  material  and  subject  factors  and  is  generated  in 
such  a manner  that  error  variance  is  basically  eliminated, 
there  nevertheless  exists  a significant  problem. 

Bormuth  (1967b)  indicates  that  "the  difficulty  of 
a text  should  be  reported  in  terms  that  make  clear  how 
appropriate  the  text  is  for  a given  individual  or  group" 

(p.  21).  In  determining  how  appropriate  the  reading 
selection  is,  one  simply  reports  how  many  individuals 
are  able  to  score  at  or  above  some  predetermined  criterion 
level.  To  do  this,  however,  requires  that  cloze  tests 
have  a criterion  score  that  represents  an  acceptable 
level  of  understanding. 

The  most  obvious  method  of  establishing  a criterion 
score  for  the  cloze  test  is  to  adopt  f.  score  traditionally 
used  and  then  determine  what  cloze  score  is  comparable 
with  that  criterion  score.  Reputable  reading  authorities 
have  suggested  that  reading  selections  are  suitable  for 
supervised  study  if  students  can  score  75  percent,  and 
selections  are  appropriate  for  unsupervised  study  if  a 
score  of  90  percent  can  be  achieved.  Unfortunately,  as  has 
been  pointed  out,  however,  no  one  has  ever  offered  any 
evidence  supporting  the  validity  of  such  scores,  and  it  is 
therefore  not  appropriate  to  use  them  in  establishing 
objective  cloze  criterion  scores. 
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An  Alternative  Cloze  Criterion  Score 

Because  traditional  scores  have  been  established 
using  subjective  rather  than  objective  standards  and  are 
therefore  most  suspect,  where  can  one  turn  in  order  to 
determine  meaningful  cloze  criterion  scores?  In  carefully 
analyzing  the  problem,  it  is  immediately  apparent  that 
a most  important  first  step  is  to  determine  what  is  act- 
ually desired  from  reading.  After  making  such  a deter- 
mination, the  task,  simply  stated,  should  be  nothing  more 
than  establishing  the  relationship  between  what  is  wanted 
from  reading  and  the  scores  achieved  by  administering 
the  cloze  test. 

While  there  are  numerous  desirable  results  from 
reading,  both  cognitive  'and  affective,  one  of  the  most 
desirable  is  that  the  reader  actually  gain  some  factual 
information  as  a result  of  having  read  a given  selection. 
If,  as  a result  of  reading,  information  is  gained  by  the 
reader,  it  should  be  readily  apparent  that  at  least  the 
following  significant  educational  aims  can  be  achieved: 

1)  Overall  curricular  objectives  can  be  better  met;  2) 
Students  can  have  a much  greater  opportunity  to  be  edu- 
cationally rewarded  and  satisfied;  3)  At  a minimum,  chances 
for  student  academic  survival  can  be  greatly  enhanced. 

In  establishing  meaningful  cloze  criterion  scores,  there- 
fore, it  would  appear  most  appropriate  simply  to  determine 
the  relationship  between  actual  information  gained  and 
cloze  scores.  More  specifically,  it  would  seem  most 


18 


valuable  if  one  could  determine  which  individual  score  on 
cloze  tests,  in  general,  reflects  that  point  at  which 
readers  of  German  actually  begin  gaining  substantial  factual 
information.  By  knowing  such  a score,  of  course,  foreign 
language  teachers  could  make  objective,  meaningful  state- 
ments concerning  the  appropriateness,  with  regard  to  dif- 
ficulty, of  any  reading  selection. 

The  Best  Means  for  Measuring  Information  Gain 

How  does  one  proceed  in  establishing  which  cloze 
score  reflects  that  point,  at  which  readers  begin  gleaning 
substantial  information?  As  an  initial  step,  one  must 
determine  how  best  to  measure  information  gain.  According 
to  Bormuth  (1967b),  the  ultimate  test  of  assessing  how 
much  information  has  been  gained  is  to  subject  the  reader 
to  the  environment  referred  to  in  the  reading  selection 
and  see  whether  he  or  she  behaves  appropriately.  Such 
a procedure,  however,  is  at  best  inconvenient,  and  in 
most  cases  impossible,  because  it  would  involve  recreating 
events  such  as  calamities  and  catastrophies  simply  to 
test  a reader's  knowledge  of  the  events  or  situations. 

As  alternatives,  therefore,  educators  have  resorted  to 
procedures  such  as  having  students  write  essay  examina- 
tions, give  oral  reports,  or  answer  objective  questions, 
either  completion-type  or  multiple-choice,  over  the 
assigned  reading  material. 

Although  all  three  of  the  above-mentioned  testing 
procedures  are  used  in  education  to  assess  how  much 
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information  the  reader  has  gained,  which  is  most  ap- 
propriate to  be  used  in  comparing  information  gain  with 
cloze  scores  and  thereby  establishing  objective  and  rational 
cloze  criterion  scores?  In  answering  this  question,  the 
following  should  be  considered: 

Stated  most  simplistically,  any  reading  selection 
can  be  viewed  as  nothing  more  than  a list  of  content- 
related  sentences.  Unlike  essay  examinations  or  oral 
reports,  conventional  objective  tests,  in  their  simplest 
form,  are  constructed  by  drawing  a predetermined  number 
of  selected  sentences  from  the  list  and  transforming  them 
into  questions  to  be  answered  either  with  a short  sentence 
(completion- type)  or  by  selecting  from  several  choices 
the  correct  answer  (multiple-choice).  In  this  situation, 
transforming  the  sentence  into  a question  has  the  effect 
of  deleting  that  portion  of  the  sentence  that  becomes 
the  correct  response  to  the  question.  In  answering  the 
question,  therefore,  one  simply  supplies  that  segment  of 
the  sentence  that  was  deleted  by  the  transformation. 

As  a most  elementary  example,  if  "the  boy"  in  the  sentence 
"The  boy  threw  the  rock  into  the  pond."  is  considered  to 
be  content  significant  and  is  therefore  selected  from 
a passage  to  serve  as  a test  question,  the  sentence  is 
transformed  into  n question  by  simply  replacing  "the  boy" 
with  the  interrogative  "who,"  which  results  in  the 
formation  of  the  question  "Who  threw  the  rock  into  the 
pond?".  In  n eomplet ion-typo  objective  test,  the  answer 


20 

would  be  "the  boy";  in  a multiple-choice  test,  "the  boy" 
would  be  included  with  two  or  three  distractors.  In  either 
case,  however,  the  answer  would  be  provided  by  supplying 
the  word  that  had  been  deleted  as  a result  of  transforming 
the  sentence  into  a question. 

From  the  foregoing,  it  is  readily  apparent  that  both  the 
cloze  test  and  the  conventional  objective  test  are  con- 
structed by  generally  like  manipulations  and  therefore 
demonstrate  at  least  a formal  similarity.  Because  of  this 
similarity,  the  objective  test  is  the  test  type  best  suited 
for  establishing  the  relationship  between  information  gain 
and  cloze  scores  and  thereby  determining  meaningful  cloze 
criterion  scores. 

An  Objective  Test  Weakness 

In  determining  the  relationship  between  the  cloze 
test  and  a conventional  objective  test  in  order  to  estab- 
lish cloze  criterion  scores,  one  major  problem  becomes 
immediately  apparent.  Although  the  two  types  of  tests 
demonstrate  formal  similarity,  a chief  distinction  between 
the  two  is  found  in  the  method  employed  to  select  the 
items  for  use  in  the  test.  As  has  already  been  pointed 
out,  cloze  tests  are  constructed  by  mechanically  and 
systematically  deleting  the  words  in  a reading  selection 
according  to  a completely  replicable  set  of  predetermined 
rules.  Conventional  objective  tests,  on  the  other  hand, 
arc  generated  in  a very  subjective  manner,  being  greatly 
influenced  by  the  biases  of  the  individual  test  writer, 


whose  rationale  for  constructing  the  various  test  items 
is  generally  based  upon  personal  feelings  about  what  he 
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or  she  calls  "important  knowledge."  In  assessing  the 
relationship  between  the  cloze  test  and  a conventional 
objective  test  in  order  to  establish  a rational  cloze 
criterion  score  based  on  information  gain,  it  is  extremely 
important  that  not  just  the  cloze  test,  but  that  both 
tests  be  completely  objective,  unbiased,  and  free  from 
all  subjective  considerations.  Only  when  this  is  ac- 
complished may  the  test  items  then  be  regarded  as  the 
property  of  the  reading  selection  itself  with  no  other 
factors  systematically  influencing  the  tests,  and  only 
then  can  one  be  sure  that  all  the  findings  will  be  unbiased 
and  therefore  valid  and  general izable . 

Objective 

Despite  the  fact  that  reading  ultimately  determines 
successful  foreign  language  learning  more  than  any  other 
of  the  four  language  skills,  assessing  the  difficulty 
level  of  German  reading  selections  in  order  to  choose 
appropriate  reading  passages  has  proven  to  be  a rather 
arbitrary  and  unexplicit  process.  Of  the  three  procedures 
presently  designed  to  measure  reading  difficulty  (read- 
ability formulas,  direct  tests  of  reading  comprehension, 
and  cloze  tests),  only  the  cloze  procedure  has  been  shown 
to  have  definite  promise  because  it  alone  denis  with  both 
material  and  subject  factors,  is  unbiased  by  the  test 
writer,  and  can  be  easily  operationally  replicated. 


Nevertheless,  failure  by  reading  experts  to  establish 
objective  guidelines  for  interpreting  cloze  scores  has 
largely  invalidated  the  cloze  procedure  as  a tool  for 
accurately  determining  the  difficulty  level  of  German 
prose  reading  selections. 

The  overall  goal  of  this  investigation,  therefore, 
is  to  establish  empirically  a meaningful  and  valid  cloze 
criterion  score  that  1)  objectively  reflects  passage  dif- 
ficulty and  2)  will  be  valid  for  any  German  prose  reading 
selection  to  which  the  cloze  test  can  be  applied.  Because 
actually  acquiring  information  is  one  of  the  most  desirable 
outcomes  of  reading,  the  cloze  criterion  score  to  be 
established  will  be  based  on  information  gain.  In  this 
regard,  the  objective  will  be  to  determine  that  score  on 
the  cloze  test  at  which  nonnative  readers  of  German  begin 
to  gain  substantial  information  as  a result  of  reading. 
Expanding  on  extensive  cloze  research  carried  out  by 
Bormuth  (1971)  in  English,  this  investigation  will  seek 
to  establish  a cloze  criterion  score  based  on  information 
gain  by  examining  and  establishing  the  relationship 
between  the  cloze  test  and  the  objective  test,  an  accepted 
measure  of  information  gain  that  has  a demonstrated  formal 
similarity  to  the  cloze  test.  More  specifically,  the 
cloze  scores  n carefully  chosen  sample  of  college  German 
students  achieve  over  a selected  piece  of  German  prose 
will  be  empirically  compared  and  analyzed  with  the  objective 
test  scores  carefully  matched  counterparts  achieve  after 
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reading  the  same  selected  piece  of  German  prose. 

In  order  to  control  subjectivity  and  individual  test 
writer  biases  and  thereby  ensure  that  the  cloze  criterion 
score  established  as  a result  of  this  research  is  both 
valid  and  generalizable  to  any  German  prose  reading  pas- 
sage to  which  the  cloze  test  might  be  applied,  it  is  es- 
sential that  both  the  cloze  test  and  the  objective  test 
be  created  following  operationally  replicable  procedures. 
Although  cloze  tests  are,  by  nature,  replicable,  such  is 
not  the  case  with  objective  tests.  Therefore,  there  arises 
the  requirement  for  generating  an  objective  test-writing 
algorithm  to  ensure  the  objectivity  and  replicability  of 
the  conventional  objective  test.  Using  ideas  set  forth 
primarily  by  Bormuth  (1970)  and  Finn  (1973),  but  altered 
to  apply  to  a German  prose  reading  selection,  such  an 
algorithm  will  be  generated  as  part  of  this  research 
project.  Details  of  the  rules  used  to  generate  the  al- 
gorithm will  be  discussed  in  Chapter  III.  In  general, 
however,  the  algorithm  is  designed  to  generate  unbiased 
and  operationally  replicable,  short-answer,  completion- 
type  objective  test  questions  for  use  in  this  research 
project.  The  completion- type  objective  test  was  chosen 
over  the  multiple-choice  type  based  on  research  reported 
by  Bormuth  (1971),  which  indicated  that  greater  con- 
formity to  the  actual  difficulty  of  the  reading  selection 
itself  could  be  achieved  by  using  short -answer , completion- 
type  test  questions. 
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Based  on  Bormuth's  (1962  and  1971)  initial  findings 
in  English,  it  is  anticipated  that  the  relationship  between 
the  two  kinds  of  tests  (cloze  and  objective)  in  this  re- 
search will  likely  he  nonlinear.  In  discussing  the  cause 
for  this  nonlinearity,  Bormuth  (1971)  indicates  that  ac- 
cording to  his  findings,  the  cloze  test,  although  similar 
to  the  conventional  objective  test,  appears  to  be  both 
more  difficult  as  well  as  capable  of  discriminating  over 
a broader  range  of  reading  abilities  than  does  the  objec- 
tive test.  For  example,  on  an  easy  reading  selection, 
objective  test  score  distributions  frequently  show  ceiling 
effects  due  to  large  numbers  of  students  obtaining  perfect 
or  very  high  scores.  On  the  other  hand,  on  difficult 
passages,  objective  test  score  distributions  often  exhibit 
flooring  effects  as  a result  of  some  students  scoring  at 
or  near  zero.  While  cloze  scores  also  exhibit  flooring 
effects,  they  seem  to  do  so  to  a lesser  degree,  and  rarely, 
if  ever,  do  they  show  ceiling  effects.  According  to 
Bormuth  (1971),  these  facts  suggest  that  there  is  a broader 
range  of  skills  involved  in  reading  than  is  normally 
identified  and  measured  by  objective  tests.  This  also 
suggests  that  the  cloze  test  is  capable  of  measuring  both 
those  skills  that  are  so  simple  and  easy  that  they  fall 
below  the  lower  limits  of  the  objective  test  as  well  as 
those  skills  that  arc  so  complex  and  difficult  that  they 
fall  above  the  upper  limits  of  the  objective  test.  Because 
of  this  greater  discriminating  capability  of  the  cloze 
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test,  it  can  therefore  theoretically  be  anticipated  that 
in  the  lower  range  of  cloze  scores,  information  gain 
scores,  as  reflected  by  the  objective  test,  will  show 
little  increase  as  cloze  scores  increase.  This,  because 
some  of  the  items  in  the  cloze  test  are  easier  than  the 
easiest  of  those  in  the  objective  test.  Therefore,  the 
objective  test  will  theoretically  fail  to  discriminate 
among  students  scoring  in  the  lower  ranges  of  the  cloze 
tests,  which  will  cause  the  slope  of  any  plotted  line  to 
stand  at  or  near  zero.  When  scores  fall  into  the  higher 
ranges  on  the  cloze  test,  however,  increases  in  the  objec 
tive  scores  will  also  be  observable  because  the  two  tests 
will  both  be  discriminating  in  that  range.  It  is  at  this 
point  that  the  slope  of  the  plotted  line  will  approach 
linearity. 

As  might  be  expected,  that  point  at  which  both  the 
cloze  scores  and  the  objective  test  scores  begin  signi- 
ficantly rising  together  can  be  viewed  as  that  point  at 
which  the  student  begins  to  gain  substantial  information 
as  a result  of  reading  as  reflected  both  by  cloze  and 
objective  test  scores.  This  point,  of  course,  can  be 
used  justifiably  as  a rational  and  objective  cloze  cri- 
terion score  because  it  reflects  that  level  on  the  cloze 
test  that  students  must  achieve  in  order  to  begin  gaining 
substantial  information  as  a result  of  reading. 
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Summary  and  Statement  of  Research  Questions 

In  achieving  general  academic  success,  reading  has 
been  identified  as  one  of  the  most  important  skills. 

Indeed,  no  comprehensive  foreign  language  program  is  con- 
ceivable without  ultimately  involving  large  quantities 
of  reading.  A most  important  factor  in  successful  reading, 
however,  is  that  the  student  be  given  selections  whose 

difficulty  level  is  within  his  or  her  range  of  reading 

1 uy  to 

comprehension.  Little  in  education  is  more  frustrating 
than  struggling  with  reading  material  far  too  advanced 
for  one's  capabilities. 

At  the  present  time,  the  assessment  of  foreign 
language  reading  difficulty,  if  objectively  assessed  at 
all,  is  determined  generally  either  by  applying  some  type 
of  readability  formula  to  the  passage,  or  by  directly 
testing  the  reader  on  the  reading  selection  in  question. 
Recently,  however,  the  validity  of  such  methods  for  asses- 
sing reading  difficulty  has  been  so  seriously  chal- 
lenged that  the  value  of  such  procedures  as  reading  asses- 
sment tools  is  seriously  questioned.  As  a result,  there- 
fore, educators  have  been  looking  for  a different,  more 
valid  means  by  which  to  assess  reading  difficulty. 

A procedure  that  has  demonstrated  great  promise 
because  of  its  objectivity,  replicability,  and  ability 
to  take  into  account  both  material  and  subject  factors 
affecting  readability  is  the  cloze  test.  Nevertheless, 
a prime  hurdle  in  effectively  using  the  cloze  test  ns 
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a means  of  determining  the  difficulty  of  reading  selections 
has  centered  around  the  establishment  of  meaningful  and 
valid  cloze  criterion  scores.  The  obvious  solution  of 
establishing  which  cloze  score  equates  to  the  75  percent 
level  of  a conventional  objective  test,  a standard  es- 
tablished and  accepted  by  most  reading  authorities,  has 
been  rejected  because  such  percentage  scores  have  proven 
to  be  arbitrary,  subjective,  and  highly  questionable. 

As  a more  meaningful  and  objective  alternative  to 
the  above,  it  has  been  decided  to  explore  the  possibility 
of  comparing  the  relationship  of  the  cloze  test  to  what 
is  actually  most  desired  from  reading,  i.e.,  the  gleaning 
of  factual  information.  In  making  this  comparison,  German 
prose  cloze-test  scores  obtained  from  carefully  matched, 
college- level , nonnative  speakers  of  German  are  to  be 
compared  with  scores  that  other  matched,  college-level, 
nonnative  speakers  of  German  achieve  on  a specially  prepared, 
Operationally  replicable,  short-answer  completion  test 
constructed  over  the  same  piece  of  German  prose.  As  a 
result  of  this  comparison,  it  is  hoped  to  be  able  to  es- 
tablish an  objective,  meaningful  cloze  criterion  score 
that  can  be  applied  to  any  German  prose  reading  selection, 
and  that  does,  in  fact ,#  ref lect  that  cloze  level  at  which 
nonnative  readers  of  German  prose  begin  to  gain  substan- 
tial information  as  a result  of  reading. 

The  purpose  of  this  research,  therefore,  is  to  de- 
termine tiie  answers  to  the  following  questions: 


1.  Is  there  a relationship  between  the  cloze  test 

and  information  gain,  as  measured  by  a specially  constructed 
short-answer  completion  test,  when  both  test  types  are 
prepared  from  a single  German  prose  reading  selection, 
selectively  administered  to  a carefully  chosen  group  of 
col  lege- 1 evcl  nonnative  speakers  of  German,  and  then 
statistically  analyzed? 

2.  If  there  is  a relationship  between  the  two  test 
types,  will  it  be  nonlinear,  as  Bormuth's  work  in  English 
suggests,  when  applied  to  nonnative  speakers  of  German? 

3.  If  the  relationship  between  the  two  test  types 
is  nonlinear,  can  a rational,  objective  cloze  criterion 
score  that  is  valid  for  any  German  prose  reading  selection 
be  identified,  which  reflects  that  point  on  the  cloze 
test  at  which  nonnative  readers  of  German  begin  gaining 
substantial  information  as  a result  of  reading  German 
prose? 


Overview 


The  balance  of  this  research  report  is  divided  into 
four  chapters  followed  by  the  appendices  and  bibliography. 

In  Chapter  II,  relevant  literature  is  reviewed.  Chapter  III 
describes  in  detail  the  instruments  and  procedures  employed 
in  this  research.  In  Chapter  IV,  the  statistical  analysis 
and  findings  are  presented  and  discussed.  Chapter  V provide 
a summary  of  the  study  including  a discussion  of  the  limi- 
tations, conclusions,  implications,  and  specific  recom- 
mendations for  additional  research. 
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CHAPTER  II 

REVIEW  OF  LITERATURE 

Introduction 

Objective 

The  purpose  of  this  research  project  is  to  establish 
an  objective,  unbiased,  and  meaningful  cloze  test  criterion 
score  that  can  be  used  by  foreign  language  teachers  as 
a basis  ior  choosing  appropriately  difficult  German  prose 
reading  selections.  To  accomplish  this  goal,  this  research 
study  has  been  designed  to  investigate  the  relationship 
between  the  cloze  test  and  information  gain,  as  measured 
by  an  operationally  replicable,  short-answer,  completion- 
type  objective  test.  The  objective  is  to  establish  at  which 
point  on  the  cloze  test  do  r.onnative  readers  of  German 
actually  begin  gaining  substantial  factual  information. 

It  is  therefore  appropriate  to  address  the  following  areas 
of  concentration:  1)  The  cloze  test  as  a measuring  instru- 
ment; 2)  The  cloze  test  and  foreign  language  research; 

3)  the  cloze  test  and  criterion  scores. 
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The  Cloze  Test  as  a Measuring  Instrument 
Cloze  Development 

Since  its  introduction  by  Wilson  Taylor  in  1953, 
the  cloze  test  has  stimulated  considerable  interest  and 
research  activity.  Originally  intended  by  Taylor  to  be 
a "new  and  better"  way  of  assessing  the  readability  of 
printed  English  prose,  its  utility  in  a variety  of  other 
research  areas  quickly  became  apparent.  After  two  years 
of  experimentation,  Taylor  (1956)  indicated  that  "the 
cloze  method  appears  to  possess  a very  large  number  of 
potential  research  uses,  both  practical  and  theoretical" 

(p.  42). 

In  1957  Rankin  completed  his  doctoral  dissertation 
on  the  cloze  procedure  and  in  1959  wrote  the  first  review 
of  research  on  studies  using  the  cloze  method.  Although 
Rankin's  review  referred  to  only  a handful  of  studies 
using  the  cloze  test,  the  procedure  subsequently  began  to 
grow  rapidly  in  popularity  as  a research  tool.  In  the 
1960s  much  research  and  many  doctoral  dissertations  covering 
a wide  variety  of  cloze- related  topics  were  completed, 
and  in  1968  the  National  Reading  Conference  conducted 
a cloze  symposium.  Since  that  time  the  cloze  test  has 
received  ever-increasing  attention  from  researchers. 

Cloze  Uses 

At  the  present,  research  involving  the  cloze  procedure 
is  generally  divided  into  three  broad  categories:  1)  re- 
search investigating  cloze  methodological  considerations 


31 


i 


in  order  to  improve  the  validity  and  reliability  of 
findings;  2)  research  in  which  the  cloze  test  is  examined 
as  a reliable  and  valid  measuring  device;  and  3)  research 
in  which  the  cloze  test  is  examined  for  its  utility  as  a 
teaching  technique.  To  date  the  majority  of  research 
has  been  directed  toward  examining  the  cloze  procedure 
in  its  role  as  a device  for  measurement. 

As  a measuring  instrument,  the  cloze  test  has  been 
used  widely  to  study  the  reading  communication  process, 
both  from  the  standpoint  of  the  message  and  its  character 
istics  (readability)  as  well  as  from  the  standpoint  of 
the  message  consumer  (comprehension).  In  this  regard, 
it  has  been  used:  1)  to  assess  and  predict  readability 
at  all  levels,  in  a variety  of  ways,  and  in  various  lan- 
guages; 2)  to  probe  the  difficulties  readers  encounter 
in  prose;  3)  to  isolate  precise  linguistic  variables 
associated  with  readability;  4)  to  measure  both  general 
and  specific  reading  comprehension;  5)  to  uncover  certain 
personality  and  physical  disorders;  6)  to  evaluate  aural 
comprehension;  and  7)  to  assess  verbal  apptitudes.  As 
can  be  seen  from  only  this  partial  list  of  cloze  appli- 
cations as  a measurement  tool,  a review  of  all  pertinent 
literature  related  to  the  various  facets  of  just  measure- 
ment is  far  beyond  the  scope  of  this  project.  In  fact, 
in  discussing  the  widespread  use  of  the  cloze  test, 

Oiler  (1973)  says  that  it  has  "been  constructed  in  so 
many  ways  and  for  so  many  purposes  that  an  overview  of 
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the  entire  scope  of  literature  on  the  subject  is  chal- 
lenging to  the  imagination  not  to  mention  the  memory" 

(p.  105). 

Pertinent  Reviews 

Because  so  much  cloze  research  has  been  conducted 
during  the  past  15-20  years,  in  this  work  it  is  necessary 
only  to  point  out  that  many  valuable  works  devoted  pri- 
marily to  reviewing  cloze  literature  have  been  published 
since  Rankin's  (1959)  initial  review.  Those  most  perti- 
nent and  comprehensive  reviews  examined  in  conjunction 
with  this  research  project  include  the  following:  1) 

The  cloze  Procedure- -A  Survey  of  Research,  by  Rankin 
(1965);  2)  Cloze  Readability  Procedure,  by  Bormuth  (1967a); 
3)  A Taxonomy  of  Cloze  Research,  Part  I:  Readability  and 
Reading  Comprehension,  by  Potter  (1968);  4)  The  Cloze 
Procedure:  A Conspectus,  by  Bickley,  Ellington,  and 
Bickley  (1970);  5)  The  Cloze  Procedure:  A Survey  of  the 
Research,  by  Jongsma  (1971);  6)  A Review  of  Literature 
Related  to  the  Cloze  Procedure,  by  Fram  (1972);  7) 

A Comprehensive  Bibliography  of  the  Cloze  Procedure,  by 
Boyce  (1974);  8)  Historical  Overview  of  the  Cloze  Proce- 
dure, by  Walter  (1974);  9)  Research  with  Cloze  Procedure 
in  Measuring  the  Proficiency  of  Non-Native  Speakers  of 
English:  An  Annotated  Bibliography,  by  Oiler  (1975); 

10)  Psychol inguistic  Experiments  in  Foreign  Language 
Testing,  by  Anderson  (1976). 
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nearly  all  cloze  investigations  used  native  speakers  of 
the  language  involved,  and  "that  in  practically  all  of 
the  published  studies  the  language  used  was  English" 

(p.  107). 

Cloze  studies  involving  foreign  languages  and  non- 
native speakers  have  been  carried  out  on  a sporadic  and 
rather  infrequent  basis.  In  reviewing  foreign  language 
cloze  research.  Oiler  (1975)  says  that  although  many  lan- 
guage teachers  and  bilingual  educators  have  begun  to  in- 
vestigate the  cloze  procedure,  "it  is  probably  true, 


nonetheless,  that  the  technique  has  been  vastly  underex- 
ploited to  date"  (p.  1).  Although  some  work  has  been 
done  examining  primarily  methodological  considerations 
(Oiler,  1972b,  for  example),  most  foreign  language  cloze 
research  has  had  as  its  ultimate  objective  the  examination 
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of  the  cloze  procedure's  reliability  and  validity  as  a 
measuring  device.  In  this  regard,  two  areas  have  been 
given  considerable  attention:  1)  measuring  global  foreign 
language  proficiency,  which  is  of  less  importance  to  this 
study,  and  2)  measuring  foreign  language  readability  and/ 
or  reading  comprehension. 
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Global  Foreign  Language  Proficiency 

At  the  present  time,  considerable  emphasis  is  being 
placed  on  the  cloze  procedure  as  a tool  for  assessing 
foreign  language  proficiency.  First  to  address  this  topic 
were  Carroll,  Carton,  and  Wilds  (1959).,  who  according 
to  Rankin  (1965),  concluded  that  the  cloze  test  could 
be  used  to  measure  group  differences  in  foreign  language 
competence  (German  and  French)  but  was  not  suitable  for 
assessing  individual  differences  because  "the  tests 
tended  to  have  low  reliability  and  were  heavily  affected 
by  various  sources  of  extraneous  cognitive  factors  which 
are  independent  of  competence  in  a foreign  language" 

(p.  140).  The  validity  of  these  findings  is  suspect, 
however,  for  at  least  three  reasons:  1)  In  preparing  the 
cloze  test,  the  random  deletion  procedure  was  not  used. 

2)  According  to  Oiler  (1975),  the  study  utilized  "a  very 
small  sample  for  many  df  its  generalizations"  (p . 20). 

3)  Instead  of  using  a proficiency  measure  for  validating, 
the  Modern  Language  Aptitude  Test  (MLAT)  was  used. 

Since  Carroll,  Carton  and  Wild's  research  in  1959, 
assessing  foreign  language  proficiency,  especially  in 
English-as-a-second- language  (ESL)  situations,  has  become 
very  popular.  In  this  regard,  Jones  (1977)  indicates 
that  Oiler  (1972a,  1975,  1976a,  1976h)  has  conducted 
extensive  research  and  that  the  cloze  procedure  "has  been 
shown  to  he  an  efficient.,  valid  and  reliable  method  of 
measuring  language  proficiency"  (p.  257).  Nevertheless, 
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although  cloze  research  in  ESL  proficiency  has  been 
widely  investigated,  especially  by  Oiler,  cloze  investi- 
gations designed  to  measure  the  foreign  language  profi- 
ciency of  native  speakers  of  English  have  been  almost 
nonexistent.  Since  Carroll,  Carton,  and  Wild's  original 
study  in  1959,  this  author  was  able  to  uncover  only  two 
other  published  cloze  research  projects  (Teitelbaum, 
Edwards,  and  Hudson,  1975;  and  Briere  et  al,  1977)  and 
two  recently  completed,  but  as  yet  unpublished  cloze  in- 
vestigations (Clausing  and  Lange,  1977;  and  Hanzeli,  1977) 
designed  to  measure  the  foreign  language  proficiency  of 
native  speakers  of  English. 

Foreign  Language  Readability 

In  addition  to  using  the  cloze  test  to  measure 
foreign  language  proficiency,  limited  interest  has  also 
been  expressed  in  cloze  measurement  of  foreign  language 
readability  and/or  reading  comprehension.  In  reviewing 
foreign  language  readability  research,  only  four  studies 
were  detected: 

1)  Taylor  (1954)  was  the  first  researcher  to  apply 
the  cloze  technique  to  a foreign  language.  Using  Korean, 
because  it  was  a language  syntactically  and  symbologically 
totally  unrelated  to  English,  Taylor  mailed  three  cloze 
tests  to  a sample  of  native  Korean  college  students 
located  throughout  the  United  States.  Prior  to  mailing, 
each  passage  was  subjectively  judged  as  easy,  medium, 
or  difficult.  Although  only  about  21  percent  of  the 
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passages  were  returned,  analysis  of  the  cloze  scores 
ranked  the  three  passages  in  the  same  order  as  the  initial 
"a  priori,"  subjective  rankings  had  placed  them. 

2)  Three  years  later,  Shiha  (1957)  found  the  cloze 
procedure  to  be  applicable  to  Japanese.  Using  eight 
Japanese  passages  that  had  been  ranked  by  three  judges, 
Shiba  reported  a correlation  of  .83  between  cloze  scores 
achieved  by  native  Japanese  and  subjective  judges'  ratings. 

3)  Anderson  (1976)  reports  that  according  to  research 
carried  out  by  Klare,  Sinaiko,  and  Stolurow  (1971),  the 
cloze  procedure  has  proven  to  be  a valid  and  reliable 
tool  for  discriminating  between  good  and  bad  translations 
of  English  into  Vietnamese. 

4)  In  the  only  known  cloze  readability  study  in  which 
nonnative  speakers  of  the  foreign  language  were  involved, 
Kerr  (1968),  utilizing  nonnative  speakers  using  French 
reading  materials,  replicated  Taylor's  initial  studies. 

In  Kerr's  investigation  three  passages  were  initially 
subjectively  ranked  by  experienced  French  teachers.  Three 
different  forms  of  the  cloze  were  then  prepared  over 
each  passage,  in  which  either  every  fifth,  every  tenth, 
or  random  ten  percent  deletions  were  made.  The  three 
versions  were  then  administered  to  Australian  students 
studying  French.  Despite  the  different  ways  in  which  the 
cloze  tests  were  prepared,  there  was  a perfect  correspon- 
dence between  teacher  ratings  and  cloze  score. 
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Foreign  Language  Reading  Comprehension 

Only  one  study  devoted  solely  to  foreign  language 
reading  comprehension  was  detected.  Using  foreign  students 
studying  at  the  University  of  Florida,  Friedman  (1964) 
found  that  cloze  scores  on  20  English  reading  selections 
correlated  significantly  with  the  Metropolitan  Achieve- 
ment Test. 

The  most  comprehensive  foreign  language  cloze  re- 
search utilizing  the  cloze  procedure  as  a measure  of  \ 

both  readability  and  reading  comprehens ion  was  reported 
by  the  Australian  Anderson  (1976).  As  part  of  an  ongoing  ; 

project,  Anderson  conducted  a series  of  ten  experiments 
designed  to  evaluate  various  methodological  considerations 
as  well  as  the  validity  and  reliability  of  the  cloze  test 
as  a measure  of  readability  and  reading  comprehension. 

Subjects  for  the  first  eight  experiments  were  second- 
language  learners  from  Papua,  New  Guinea  and  Fiji,  and 
all  reading  materials  were  in  English.  For  the  last  two 
experiments,  subjects  were  drawn  from  Malay-  and  Chinese- 
medium  schools  in  Singapore,  and  all  reading  materials 
were  in  Malay  and  Chinese  respectively.  In  summarizing 
the  overall  findings  of  the  research,  Anderson  (1976) 
says  the  following: 

To  conclude,  evidence  from  the  ten  experiments 
has  indicated  the  usefulness  of  cloze  procedure 
as  a method  of  measuring  reading  difficulty  and  of 
testing  reading  comprehension  with  subjects  for 
whom  English  is  a second  or  foreign  language,  and 
to  a less  extent,  with  native  speakers  of  Malay  and 
Chinese.  The  wider  application  of  cloze  procedure 
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to  cross-cultural  and  foreign  language  testing  is 
a development  on  the  horizon  which  offers  exciting 
prospects  (p.  132). 

As  was  the  case  in  cloze  research  intended  to  assess 
foreign  language  proficiency,  few  cloze  studies  designed 
to  measure  foreign  language  readability  or  reading  comp- 
rehension have  involved  native  speakers  of  English  using 
a foreign  language.  Except  for  Kerr's  (1968)  research, 
no  other  evidence  of  such  investigations  was  detected. 

The  Cloze  Test  and  Criterion  Scores 
Although  considerable  research  has  been  carried  out 
to  assess  the  value  of  the  cloze  procedure  as  a tool  for 
measuring  reading  comprehension,  especially  in  English, 
the  problem  of  objectively  interpreting  cloze  test  results 
has  continually  perplexed  investigators.  So  serious  has 
the  barrier  been  that  Walter  (1974)  indicates:  "The 
problem  of  how  to  interpret  cloze  test  scores  has  plagued 
researchers  more  than  any  other  facet  of  the  cloze  pro- 
cedure" (p.  23).  . 

Early  Approaches 

In  initial  cloze  research,  investigators  used  raw 
scores  to  interpret  results.  This  method  is  of  little 
value,  however,  because  cloze  tests  vary  in  length.  There- 
fore, at  best,  the  raw  score  can  only  have  meaning  for 
each  individual  test. 

In  an  attempt  to  circumvent  the  problem  created  by 
using  raw  scores,  many  resea rehers  began  converting  raw 
scores  to  percentages.  Although  preferable  to  the  raw 
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score  because  it  allows  comparisons  between  cloze  tests 
of  different  lengths,  the  cloze  percentage  score  is  only 
slightly  better  than  the  raw  score  because  it  also  does 
nothing  more  than  reflect  a rank  ordering.  For  example, 
if  student  X scores  50  percent  and  student  Y scores  40 
percent  on  a cloze  test  intended  to  measure  reading 
comprehension,  one  can  only  safely  assume  that  student 
X performed  better  on  the  cloze  test  than  did  student  Y. 
What  is  not  known,  however,  is  how  precisely  each  student 
comprehended  the  selection. 

Finding  Equivalent  Scores 

Because  neither  the  raw  score  nor  the  percentage 
score  was  a useful  and  meaningful  cloze  criterion,  investi- 
gators began  examining  how  the  dilema  could  be  resolved. 

A logical  and  direct  approach  to  the  problem  seemed  to 
be  to  adopt  a cloze  criterion  score  that  was  equivalent 
to  a performance  level  traditionally  accepted.  For  years 
reading  authorities  have  accepted  75  percent  and  90  per- 
cent as  adequate  supervised  and  unsupervised  performance 
levels  in  assessing  reading  comprehension.  Because  of 
this  broad  acceptance,  Bormuth  (1967c)  therefore  decided 
to  determine  the  cloze  scores  that  were  equivalent  to  the 
traditional  75  percent  and  90  percent  rending  comprehension 
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levels . 

Selecting  100  students  enrolled  in  grades  four  and 
five,  Bormuth  used  a regression  prediction  equation  to 
match  cloze  test  results  with  performance  on  multiple* 
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choice  comprehension  tests  over  the  same  nine  passages. 

His  findings  indicated  that  a cloze  score  of  45  percent 
was  comparable  with  a mul t i pi e- choi ce  comprehension  score 
of  75  percent  and  that  a cloze  score  of  52  percent  was 
comparable  with  a comprehension  score  of  90  percent. 

Not  completely  satisfied  with  these  results,  however, 
the  next  year  Bormuth  (1968)  determined  to  establish 
which  cloze  scores  equated  to  the  75  percent  and  90 
percent  criteria  when  measured  with  completion-type  comp- 
rehension tests  administered  during  an  oral  reading  test. 
Taking  120  students  enrolled  in  grades  four,  five,  and 
six,  Bormuth  administered  all  four  equivalent  forms  of  the 
Gray  Oral  Reading  Paragraphs,  each  form  of  which  consists 
of  13  paragraphs  graded  according  to  difficulty.  First, 
every  subject  was  given  a cloze  test  over  two  of  the  four 
p' _^ible  paragraphs  at  each  level  of  difficulty.  Then, 
all  were  given  an  orally  administered,  completion- type 
comprehension  test  over  each  of  the  two  remaining  paragraphs 
immediately  after  having  read  each  paragraph  orally. 

The  cloze  scores  equivalent  to  the  criterion  scores  were 
established  by  finding  the  most  difficult  level  at  which 
a student  was  able  to  obtain  first  a score  of  75  percent 
and  then  a score  of  90  percent  on  the  two  sets  of  comp- 
rehension tests.  When  this  level  was  determined,  the 
two  eloze  test  scores  achieved  over  the  other  two  com- 
parable selections  were  averaged  to  determine  the  cloze 
criterion  score.  Results  of  Bormuth's  findings  in  this 
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study  reflected  that  a cloze  score  of  44  percent  was 
comparable  with  an  orally  administered,  completion-type 
comprehension  test  score  of  75  percent  and  that  a cloze 
score  of  57  percent  was  equivalent  to  the  90  percent 
level  on  that  type  of  test. 

In  explaining  the  difference  between  the  two  cloze 
criterion  scores  established  with  the  multiple-choice 
tests  as  compared  with  those  found  with  the  completion 
test  (see  Table  1),  Bormuth  (1969)  says  the  90  percent 
criterion  score  on  the  multiple-choice  test  was  most  likely 
depressed  by  a ceiling  effect.  Therefore,  the  score 
obtained  in  the  1968  study  "seems  to  be  a better  estimate 
of  the  cloze  score  comparable  to  the  90  per  cent  criterion" 

(p.  62). 

Recognizing  the  importance  of  establishing  meaning- 
ful and  objective  cloze  criterion  scores,  researchers 
have  conducted  two  investigations  to  replicate  Bormuth ' s 
1967  study.  Rankin  and  Culhane  (1969)  selected  105  students 
in  grade  five  and  compared  their  cloze  test  results  with 
multiple-choice  tests  over  the  same  five  reading  selec- 
tions. Results  indicated  that  a cloze  score  of  41  per- 
cent was  comparable  with  a multiple-choice  comprehension 
score  of  75  percent,  while  a cloze  score  of  61  percent 
equated  to  the  90  percent  comprehension  score. 

In  the  only  study  in  which  nonnative  speakers  of  the 
language  were  involved  to  establish  a meaningful  cloze 
criterion  score,  Anderson  (1976)  selected  284  elementary- 
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TABLE  1 

RESEARCH  FINDINGS  BY  BORMUTH  DESIGNED  TO  ESTABLISH  CLOZE 
CRITERION  SCORES  EQUIVALENT  TO  TRADITIONALLY 
ACCEPTED  PERFORMANCE  LEVELS 


Traditional  Scores 


75 


90 


Comparable Cloze 


Scores 


1967  Study  1968  Study 

(Multiple  Choice)  (Completion) 

45  44 


52 


57 
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school  students  from  three  separate  scnools  in  Papua, 

New  Guinea.  Using  English  passages  ordered  according  to 
difficulty,  both  a cloze  test  and  a multiple -choice  com- 
prehension test  were  constructed  over  each  passage. 
Students  were  first  administered  the  cloze  tests  and  then 
one  week  later  were  given  the  multiple-choice  tests. 

As  a result  of  regression  analysis,  Anderson  reported  that 
a cloze  score  of  44  percent  equated  to  a multiple-choice 
comprehension  score  of  75  percent.  A cloze  score  of  53 
percent  was  equivalent  to  the  90  percent  comprehens ion 
score  (sec  Table  2). 
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Two  Variations 

In  a related  attempt  to  establish  a meaningful  cloze 
criterion  score,  Alexander  (1968)  gave  365  elementary- 
level  students  cloze  tests  and  the  Ga tes -MacGin i t ie 
Reading  Tests- -Survey  D.  A statistical  analysis  that 
compared  the  results  revealed  that  a cloze  score  any  lower 
than  47  percent  resulted  in  student  frustration,  while  a 
cloze  score  of  61  percent  or  above  suggested  the  student 
could  read  the  material  independently.  These  two  per- 
centages (47  and  61),  of  course,  equate  to  Bormuth's  75 
and  90  percent  levels. 

In  a slightly  different  and  somewhat  questionable 
approach.  Ransom  (1968)  arbitrarily  established  the  cloze 
score  of  50  percent  to  represent  an  independent  reading 
level,  30  to  50  percent  to  represent  and  instructional 
level,  and  below  20  percent  to  represent  a -frustration 
level.  She  then  correlated  her  predetermined  cloze  scores 
with  the  results  of  an  Informal  Reading  Inventory.  Because 
correlations  were  high,  she  assumed  her  comparable  cloze 
scores  were  valid  for  estimating  students'  reading  levels 
from  grades  two  through  six. 

Problems  with  Equivalent  Scores 

Although  all  of  the  research  studies  mentioned  above 
are  significant  steps  toward  establishing  meaningful  and 
objective  cloze  criterion  scores,  there  are  nevertheless 
serious  inctliodological  weaknesses  that  make  their  findings 
suspect.  Disregarding  individual  weaknesses  of  design, 


< 


* 

\ 

•I 

1 


1 


45 


such  as  Ransom  (1968)  arbitrarily  selecting  her  cloze 
criterion  scores,  the  following  should  be  pointed  out: 

In  conducting  their  investigations,  all  of  these  researchers 
(Bormuth,  1967,  1968;  Alexander,  1968;  Ransom,  1968;  Rankin 
and  Culhane,  1969;  and  Anderson,  1976)  analyzed  their  cloze 
test  results  with  some  form  of  a direct  test  of  reading 
comprehension.  As  was  pointed  out  in  Chapter  I,  direct 
tests  of  reading  comprehension  have  at  least  three  basic 
weaknesses : 

1)  There  is  strong  evidence  that  such  tests  are  not 
reliable  measures  of  content  difficulty  alone  because  test 
writers  tend  to  write  questions  in  such  a fashion  that  the 
language  used  is  either  easier  or  more  difficult  than  the 
language  in  the  actual  reading  selection.  If  the  test 
reflects  anything  other  than  the  actual  difficulty  of  the 
reading  selection,  the  cloze  criterion  scores  that  are 
subsequently  established  will  be  proportionately  invalid. 

2)  The  manner  in  which  test  items  arc  first  generated 
and  then  selected  for  inclusion  into  the  reading  comprehen- 
sion test  is  suspect.  Instead  of  systematically  generating 
a population  of  possible  test  questions  in  an  operationally 
replicable  manner  that  is  open  to  investigation  and  then 
sampling  from  that  population  to  determine  the  specific 
questions,  test  questions  generally  are  created  according 
to  biases  and  preferences  of  each  individual  test  writer. 
Again,  the  result  is  a distortion  of  t he  findings.  In 
addressing  this  very  problem,  Bormuth  ( I !>*»*> ) says  of  his 
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1967  and  1968  studies  that  they  "share  the  problem  that 
the  population  of  traditional  comprehension  test  items 
used  were  not  rigorously  defined  and  then  sampled  to  compose 
the  tests  " (pp.  62-63). 

3)  There  is  no  evidence  anywhere  in  the  literature 
that  supports  the  supposition  that  the  criterion  scores 
of  75  percent  and  90  percent  are  ideal.  No  reading  expert 
has  ever  offered  any  reasoning  or  rational  support  for 
such  scores.  Instead,  these  two  figures  have  arisen  as 
a result  of  tradition--certainly  a poor  criterion  for 
selecting  meaningful  cloze  criterion  scores. 

Considering  the  basic  weaknesses  associated  with 
direct  tests  of  reading  comprehension  used  in  the  studies 
discussed  above,  it  is  evident  that  any  resultant  cloze 
criterion  scores  must  be  viewed,  at  best,  as  very  ques- 
tionable. In  discussing  three  of  the  above  studies, 

Jongsma  (1971)  points  out  that  they  are  honest  attempts 
at  building  a framework  to  help  in  making  value  judge- 
ments about  cloze  test  results.  There  are  limitations, 
however: 

They  all  are  based  on  the  notion  of  independent, 
instructional  and  f rustrational  reading  levels  and 
the  commonly  used  percentages  for  defining  such 
levels.  The  research  that  the  definitions  of  these 
levels  is  based  upon  is  far  from  convincing  or  con- 
clusive. Any  attempt  to  interpret  cloze  test  results 
on  the  basis  of  such  rationale  must  be  treated 
cautiously  (p.  20). 
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Accepting  the  fact  that  the  validity  of  cloze 
criterion  scores  is  suspect  when'  established  by  finding 
what  cloze  score  equates  to  75  percent  and  90  percent 
on  traditionally  accepted  reading  comprehension  tests, 
researchers  interested  in  determining  more  objective 
cloze  criterion  scores  began  investigating  other  avenues. 

The  groundwork  for  a more  objective  approach  to 
establishing  cloze  criterion  scores  was  laid  by  Coleman 
and  Miller  (1968)  whose  research  objective  was  to  determine 
what  level  of  cloze  passage  difficulty  resulted  in  the 
greatest  amount  of  information  gain.  Using  undergraduate 
college  students,  they  measured  information  gain  by  typing 
a reading  passage  on  a transparency  and  then  covering  the 
words  with  strips  of  tape.  When  this  was  projected  on  an 
overhead,  the  students  were  asked  to  guess  and  write  down 
the  first  word.  This  word  was  then  exposed  and  the  students 
were  then  asked  to  guess  and  write  down  the  next  word. 

After  all  words  in  the  passage  had  been  guessed  in  this 
manner,  the  tape  was  replaced  over  the  passage  and  the 
procedure  was  repeated.  The  difference  between  scores  on 
the  two  trials  was  taken  as  a measure  of  information  gain. 
Passage  difficulty  was  established  on  a matched  group  of 
subjects  who  were  administered  cloze  tests.  Of  interest 
is  the  fact  that  results  indicated  that  maximum  information 
gain  occurred  on  passages  having  difficulties  of  close 
to  44  porccnt--thc  cloze  score  Rormutli  (1967c,  1968) 
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found  to  be  equivalent  to  the  traditional  75  percent 
criterion . 

Although  Coleman  and  Miller's  research  could  be 
criticized  (MacGinitie,  1961;  Walter,  1974,  for  example) 
because  the  information  gained  may  be  overly  influenced 
by  rote  memorization,  Bormuth  (1967a)  says  that  "it  seems 
clear  that  this  study  (Coleman  and  Miller,  1968)  demon- 
strated how  a rational  approach  can  be  made  to  the  estab- 
lishment of  criterion  scores"  (p.  19), 

Bormuth 's  Approach 

Expanding  on  Coleman  and  Miller's  (1968)  belief 
that  meaningful  cloze  criterion  scores  could  be  rationally 
established  by  investigating  the  relationship  of  the  cloze 
test  to  information  gain,  Bormuth  (1971)  conducted  a 
series  of  five  studies  ultimately  designed  to  investigate 
the  relationship  of  cloze  scores  to  not  only  the  cognitive 
variable  of  information  gain,  but  to  five  other  variables 
as  well  (rate  of  reading,  willingness  to  study,  and 
preferences  for  the  subject  matter,  style,  and  level  of 
difficulty) . 

In  Bormuth's  research,  the  intent  was  not  to  estab- 
lish at  which  point  on  the  cloze  continuum  comprehension 
began  to  occur.  Instead,  his  objective  was  to  determine 
at  which  point  on  the  cloze  test  did  the  student  achieve 
maximum  comprehension.  Although  such  an  objective  is 
slightly  different  from  that  of  this  study,  the  means 
by  which  these  two  objectives  arc  accomplished  are,  in 
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this  instance,  generally  the  same.  Because  just  the  first 
three  studies  in  Bormuth's  series  deal  with  objectively 
establishing  the  relationship  between  cloze  scores  and 
information  gain,  only  they  will  be  treated  in  this  review. 
Some  Assumptions 

In  attempting  to  improve  the  objectivity  of  his 
research,  Bormuth  (1971)  established  three  assumptions 
taken  from  psychometric  theory,  which  he  felt  were  es- 
sential in  order  to  establish  meaningful  and  objective 
cloze  criterion  scores.  These  he  referred  to  as  the 
instructional  conformity,  the  behavioral  consistency, 
and  the  regression  identity  assumptions. 

The  instructional  conformity  assumption  asserts 
that  in  order  for  any  comprehension  test  made  from  a 
reading  selection  to  be  valid,  it  must  represent  only 
the  characteristics  of  the  passage  itself  and  not  some 
other  irrelevant  considerations.  This  assumption,  of 
course,  addresses  the  problem  of  writing  operationally 
replicable  comprehension  tests  that  have  not  been 
arbitrarily  manipulated  or  tampered  with  as  a result  of  the 
supposed  "good  judgement"  of  the  test  writer  himself. 

The  second  assumption,  the  behavioral  consistency 
assumption,  refers  to  the  relationship  among  the  behaviors 
measured  by  tests  that  are  of  the  same  type  but  that  are 
made  from  different  passages.  This  assumption  asserts 
that  tests  of  the  same  kind  made  from  different  reading 
passages  must  measure  the  same  type  of  behavior  on  all 
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of  the  reading  passages.  Bormuth  (1971)  says  that  this 
can  be  accomplished  "by  making  the  same  type  of  items 
for  each  passage"  (p.  29) --something  the  cloze  test  does 
but  most  traditional  reading  comprehension  tests  di.  not 
do  because  they  are  constructed  in  such  a subjective  man- 
ner. Bormuth  indicates,  however,  that  traditional  reading 
comprehension  tests  also  can  be  made  to  comply  with  the 
behavioral  consistency  assumption  by  constructing  them 
in  an  operationally  replicable  manner.  In  support  of 
this,  research  carried  out  by  Bormuth,  Manning,  Carr,  and 
Pearson  (1970)  is  cited,  in  which  the  researchers  first 
defined  large  numbers  of  classes  of  comprehension  items, 
constructed  one  item  from  each  of  a number  of  different 
paragraphs,  and  finally  demonstrated  that  items  generated 
by  the  same  rules  exhibited  behavioral  homogeneity  despite 
the  fact  that  they  were  written  from  different  passages. 

The  final  assumption,  the  regression  identity  as- 
sumption, asserts  that  a given  type  of  test  used  to  es- 
tablish a performance  criterion  must  exhibit  "parametri- 
cally identical  regressions  on  the  independent  variable, 
the  cloze  tests,  regardless  of  the  passage  from  which 
it  is  made  or  the  student  to  whom  it  is  given"  (Bormuth, 
1970,  p.  30).  This  is  most  significant  because  if  the 
regressions  for  some  type  of  comprehension  test  differ 
from  passage  to  passage,  the  cloze  criterion  scores  will 
also  differ,  and  it  will  therefore  be  impossible  to 
establish  cloze  criterion  scores  for  anything  broader  than 
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each  individual  reading  selection.  By  the  same  token, 
if  the  resultant  regressions  differ  for  various  groups 
of  readers,  different  criteria  will  have  to  be  established 
for  each  individual  group. 

Study  One 

In  his  initial  study,  Bormuth  (1971)  selected  260 
English-speaking  students  ranging  from  grade  three  through 
graduate  school.  In  the  first  of  three  separate  testing 
sessions,  the  students  were  administered  a 52-item  cloze 
test  in  order  to  form  individual  pairs  of  readers  matched 
for  reading  comprehension  ability.  To  match  the  students, 
Bormuth  ranked  them  according  to  the  size  of  their  scores 
on  the  cloze  test  and  then  matched  them  in  order,  a pair 
at  a time.  After  matching,  one  member  of  each  pair  was 
randomly  designated  member  X and  the  other  member  Y. 

Two  short  reading  passages  were  then  selected  (469 
words  and  398  words)  and  designated  passages  A and  B. 

From  each  of  these  two  passages  were  constructed  two  types 
of  tests,  a four-alternative,  multiple-choice  compre- 
hension test  (passage  A,  34  questions  and  passage  B, 

39  questions)  and  five  different  forms  of  a cloze  test 
in  which  every  fifth  word  was  deleted.  In  constructing 
five  separate  forms  of  the  cloze  test,  the  first  form  had 
words  1,  6,  11,  etc.  deleted;  the  second  form  had  words  2, 

7,  12,  etc.  deleted,  etc.  until  all  five  forms  of  the 

cloze  tost  had  been  constructed  over  each  of  the  two  passages, 

A and  B.  Five  different  forms  of  the  cloze  test  were  used 
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on  each  passage  to  counterbalance  any  effects  that  might 
arise  should  the  items  in  one  form  differ  in  difficulty 
from  the  average  difficulty  of  the  items  in  all  five  forms. 

To  ensure  this,  the  five  forms  were  randomly  assigned  to  stu- 
dents so  that  each  form  occurred  equally  often  throughout  the 
entire  study. 

At  the  second  testing  session,  all  of  the  students 
designated  member  X were  given  the  multiple-choice  test  made 
from  passage  A (without  having  first  read  passage  A)  and  one 
of  the  five  randomly  assigned  forms  of  the  cloze  test  over 
passage  B.  At  the  same  time,  the  Y members  were  administered 
the  multiple-choice  test  made  from  passage  B (without  having 
first  read  passage  B)  and  one  of  the  five  randomly  assigned 
forms  of  the  cloze  test  made  from  passage  A.  The  multiple- 
choice  tests  were  given  prior  to  allowing  the  students  to  read 
the  passages  in  order  to  establish  how  much  prioT 'khAWledge 
the  students  possessed  about  the  passage. 

In  the  final  testing  session,  conducted  about  10 
days  later,  the  X members  were  asked  to  read  passage  A 
and  then  retake  the  multiple-choice  test  over  it,  while 
the  Y members  were  doing  the  same  thing  with  passage  B. 

The  testing  sequence  followed  during  this  study  is  shown 
in  Table  3. 

In  preparing  his  data  for  analysis,  Bormuth  computed 
the  cloze  scores  over  the  two  passages  by  finding  the 
percentage  of  cloze  items  that  each  student  had  answered 
correctly.  Information  gain  scores  were  computed  by  first 
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correcting  all  multiple-choice  scores  for  guessing  and 
then  adjusting  and  subtracting  the  prereading  scores  from 
the  postreading  scores.  These  information  gain  scores 
also  were  then  expressed  as  percentage  scores. 

The  actual  analysis  consisted  of  taking  each  pair's 
cloze  and  information  gain  score  over  each  of  the  two 
reading  passages  and  comparing  them  by  means  of  regression 
analysis.  Upon  finding  that  the  two  resultant  curves  were 
very  similar,  Bormuth  combined  the  two  sets  of  data  for 
each  pair  and  reanalyzed  his  findings.  Speaking  very 
generally,  his  findings  suggested  that  students  who  scored 
no  more  than  25  percent  on  the  cloze  test  gained  little, 
if  any,  information  from  the  reading  passages.  Above  that 
point,  there  was  a sharp  and  steady  increase  in  the  amount 
of  information  gained.  According  to  Bormuth,  at  least 
three  things  made  the  findings  of  this  initial  study  some- 
what cloudy  and  questionable: 

1)  The  two  multiple-choice  tests  used  for  the  study, 
although  very  carefully  and  thoughtfully  constructed, 
were  not  systematically  generated  in  an  operationally 
replicable  manner  and  therefore  probably  reflected  some- 
thing other  than  the  actual  difficulty  of  the  two  reading 
passages.  As  a result,  both  Bormuth's  assumption  of 
instructional  conformity  and  his  assumption  of  behavioral 
consistency  were  violated  and  the  findings  became  suspect. 

2)  Only  two  reading  passages,  both  similar  in  dif- 
ficulty, were  used  in  the  study.  Because  more  passages 
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j of  varying  difficulty  might  have  significantly  altered 

the  shapes  of  the  regressions,  Bormuth  could  not  defini- 
tively say  that  this  research  confirmed  and  validated 
his  regression  identity  assumption- -something  absolutely 
essential  in  establishing  meaningful  cloze  criterion 
■ scores. 

3)  Bormuth's  analysis  of  his  findings  reflected  the 
unexpected  tendency  for  older  students  to  gain  slightly 
more  information  at  each  cloze  level.  This  suggested,  of 
course,  that,  at  least  for  children  who  were  in  the  process 
I of  developing  their  reading  skills,  a slightly  different 

cloze  criterion  score  might  have  to  be  established  for 
each  grade  level. 

Study  Two 

Recognizing  the  weaknesses  inherent  in  Study  One, 
but  also  seeing  the  great  potential  for  taking  such  an 
approach,  Bormuth  (1971)  conducted  his  second  study,  which 
was  designed  primarily  to  accomplish  two  broad  objectives: 
1)  to  provide  greater  controls  for  the  three  problems  that 
had  been  identified  in  Study  One,  and  2)  to  consider  the 
effects  of  the  students'  attitude  toward  the  reading 
1 materials,  a factor  Bormuth  felt  might  be  of  significance. 

Because  using  matched  pairs  of  students  tested 
during  several  (three)  different  sessions  had  resulted 
in  large  losses  of  cases  due  to  absences  at  one  or  more 
of  the  testing  sessions  in  Study  One,  Bormuth  designed 
' Study  Two  so  that  all  data  could  be  gathered  in  a single 
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testing  session.  To  do  this,  each  student  was  given  a 
cloze  test  over  one  passage  and  a preference-rating  scale 
and  pre-  and  postreading  complet ion  - type  tests  all  made 
from  a second  passage  of  matched  difficulty. 

I 

To  conduct  the  study,  Bormuth  selected  1320  students 
enrolled  in  grades  three  through  12.  lie  then  took  44  short 
reading  selections  equally  drawn  in  groups  of  11  from  the 
four  different  forms  of  the  Gray  Oral  Reading  Paragraphs 
(1963  edition)  and  arranged  them  according  to  difficulty 
from  very  easy  to  very  hard  so  that  there  existed  four 
separate,  but  equivalently  difficult,  reading  passages 
at  each  of  11  difficulty  levels.  From  each  of  these  44 
reading  passages  he  then  constructed  five  different  forms 
of  a cloze  test  in  which  various  combinations  of  every 
fifth  word  were  deleted.  For  example,  the  first  form  had 
words  1,  6,  11,  etc.  deleted;  the  second  form  had  words 
2,  7,  12,  etc.  deleted,  etc.  until  all  five  forms  of  the 
cloze  test  had  been  constructed  over  each  of  the  44  pas- 
sages. In  addition,  a ten-item,  completion- type  compre- 
hension test  was  constructed  covering  each  of  the  44  pas- 
sages. The  completion-type  comprehension  test  was  chosen 
over  the  multiple-choice  type  because  of  difficulty  Bormuth 
had  encountered  in  Study  One  in  writing  acceptable  alterna- 
tives that  were  unbiased  and  reflected  only  the  difficulty 
of  the  passage,  and  because  Bormuth  (1970)  had  developed 
a more  objective  complet ion-type  question  writing  procedure 
based  upon  wh-  rote  questions  constructed  from  wh-  words 
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and  phrases  such  as  who,  what,  where,  when,  what  kind  of, 
how  many,  etc.  Finally,  a seven-point  interest  scale 
ranging  from  "dislike  very  much"  to  "like  very  much"  was 
devised  to  measure  the  students'  willingness  to  study  the 
textbook  that  a passage  supposedly  represented. 

Using  all  of  the  above  materials,  test  booklets  were 
then  constructed,  each  of  which  contained  various  tests 
made  from  one  of  two  passages  of  similar  difficulty. 

Found  on  pages  two  through  five  (page  one  contained  stu- 
dent identification  data)  in  eacl  test  booklet  were  the 
following:  a complet ion-type  comprehension  test  over  one 

of  the  two  passages  (p.  2);  a cloze  test  made  from  the 
second  passage  (p.  3);  the  passage  from  which  the  first 
completion  test  on  page  two  had  been  made  and  the  rating 
scale  (p.  4);  the  same  completion-type  comprehension  test 
the  student  had  taken  on  page  two  prior  to  reading  the 
passage  (p.  5).  r , i, , 

In  assembling  these  test  booklets,  a counterbalanced 
design  was  used  in  which  the  four  passages  at  each  of 
the  11  different  difficulty  levels  were  coded  A through 
D and  then  arranged  by  pairs,  AB  through  DC,  into  11 
different  blocks  of  12  each.  In  this  sequence,  the  first 
letter  of  each  pair  designated  that  passage  to  be  use(d 
as  a cloze  test,  and  the  second  letter  identified  that 
passage  to  be  used  as  a measure  of  information  gain  (see 
Table  4).  Ten  such  blocks,  one  for  each  grade  level 
(3-12),  were  made  at  each  of  the  11  different  difficulty 
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TABLE  4 

BORMUTH'S  COUNTERBALANCED  DESIGN  USED  IN  STUDY  TWO 


AB 

AC 

AD 

BC 

BD 

CD 

BA 

CA 

DA 

CB 

DB 

DC 

levels  resulting  in  1320  test  booklets,  To  ensure  average 
difficulty,  the  five  separate  forms  of  each  cloze  test 
were  randomly  assigned  to  all  of  the  ten  blocks  at  each  of 
the  11  difficulty  levels.  In  employing  the  above  design, 
Bormuth  thereby  assured  himself  of  having  all  combinations 
of  both  test  types  at  all  11  difficulty  levels  equitably 
administered  to  all  ten  grades. 

In  analyzing  the  data  collected  from  this  study  by 
means  of  regression  analysis,  Bormuth  reported  the  fol- 
lowing: 1)  Regarding  the  necessity  for  calculating 

separate  cloze  criterion  scores  based  on  information  gain 
for  students  at  each  grade  level,  findings  were  ambiguous. 
Although  a strong  interaction  did  occur  between  the  cloze 
test  and  grade  level  when  the  wi 1 1 ingness-to-study  ratings 
were  used  as  the  dependent  variable,  there  was  only  partial 
evidence  that  came  from  a small  ancillary  study  that  sug- 
gested the  same  was  true  for  information  gain.  In  the 
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main  study,  Bormuth  (1971)  reported  that  the  interaction 
between  cloze  scores  and  information  gain  scores  was  am- 
biguous and  may  have  been  made  so  by  the  fact  "that  there 
seemed  to  be  a very  large  carry-over  effect  from  admini- 
stering the  same  test  as  both  a pre-reading  and  post-reading 
test"  (p.  77).  Therefore,  although  there  was  some  evi- 
dence that  older  students  gained  more  information  at  each 
cloze  level,  the  question  still  remained  unresolved. 

2)  Regarding  the  effects  of  using  many  passages 
having  multiple  difficulty  levels,  findings  from  Study 
Two  indicated  that  although  tests  with  relatively  few 
items  had  been  used,  the  shapes  of  the  regression  were 
most  likely  very  nearly  the  same.  Again,  however,  findings 
were  suspect  because  of  the  carry-over  effect  resulting 
from  administering  the  same  test  as  both  a prereading 
and  postreading  test.  Because  of  this,  Bormuth  (1971) 
indicated  that  the  findings  from  Study  Two  only  provided 
"indirect  evidence  that  the  regression  identity  assumption 
held  to  a reasonable  degree  for  the  information  gain 
regressions"  (p.  73).  Nevertheless,  Bormuth  continued, 
"there  seemed  to  be  adequate  grounds  for  proceeding  on 
the  assumption  that  the  regression  identity  assumption 
was  satisfied  to  a reasonable  degree  of  approximation 
by  the  tests  employed  in  this  study"  (p.  78). 

Study  Three 

In  Study  Two,  carry-over  effects  caused  from  admini- 
stering the  same  test  as  both  a prercading  and  postreading 
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comprehension  test  resulted  in  less  than  clear  findings 
with  regard  to  the  shape  of  the  regression  obtained  over 
many  passages  of  varying  difficulty.  Additionally,  questions 
were  raised  with  regard  to  both  the  preference  ratings 
that  had  been  gathered  as  well  as  the  comp  let  ion-type 
comprehension  tests  that  had  been  constructed.  As  a result, 
the  third  study  was  launched. 

Because  the  basic  design  used  in  Study  Two  had  been 
highly  efficient,  it  was  retained  in  Study  Three  with 
three  changes:  First,  instead  of  using  only  two  passages 
of  matched  difficulty,  one  for  the  cloze  test  and  the  other 
as  a basis  for  constructing  both  prereading  and  postreading 
completion-type  comprehension  tests  as  well  as  a preference 
rating  scale,  four  separate  passages  of  equal  difficulty 
were  used.  From  the  first  passage  was  constructed  the 
cloze  test.  From  the  second  came  the  prereading  compre- 
hension test.  From  the  third  was  taken  the  postreading 
comprehension  test.  And  finally,  the  fourth  passage  was 
used  as  a basis  for  administering  an  expanded  preference 
scale  in  which  each  student  was  required  to  rate  a passage 
on  ten  separate  scales  involving  subject-matter  preference, 
style,  difficulty,  wi 1 1 ingness - to-study , and  use  for  which 
the  reading  material  was  intended,  i.c.,  as  a textbook, 
as  reference  material,  or  as  voluntary  rending.  According 
to  Bormuth  (1971),  four  different  passages  of  matched 
difficulty  were  used  instead  of  two,  which  had  been  the 
case  in  Study  Two,  "in  order  to  eliminate  carry-over 
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effects  that  resulted  from  administering  the  same  test  as 
both  a pre-reading  and  a post-reading  test  within  the  short 
time  spans  involved  in  these  testing  operations"  (p.  81). 

The  second  change  in  Study  Three  involved  using  pas- 
sages believed  to  be  more  representative  of  instructional 
materials,  and  the  last  change  saw  the  implementation  of 
both  longer  reading  selections  and  comprehension  tests  in 
order  to  improve  reliability. 

In  conducting  Study  Three,  Bormuth  selected  1600 
students  enrolled  in  grades  three  through  12  in  a middle- 
class  suburban  school  system.  In  preparing  materials, 
eight  sets  of  reading  passages  representing  eight  dif- 
ferent difficulty  levels  were  constructed.  Contained 
within  each  of  the  eight  different  sets  were  four  reading 
selections  that  had  been  matched  previously  by  Bormuth 
for  cloze  difficulty.  Unlike  the  44  passages  used  in 
Study  Two,  these  32  passages  were  longer  (each  passage 
contained  a minimum  of  250  words)  and  represented  material 
specifically  selected  by  Bormuth  because  they  were  repre- 
sentative of  either  some  randomly  selected  section  of  a 
textbook  or  some  other  form  of  instructional  materials. 

As  had  been  the  case  in  the  previous  two  studies,  five 
different  forms  of  the  cloze  test  were  constructed  over 
each  of  the  passages.  As  had  been  done  in  Study  Two,  a 
completion-type  comprehension  test  was  also  made  from  each 
passage.  Instead  of  ten  short-answer  completion  items, 
however,  20  items  were  constructed.  Iixccpt  for  minor 
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modifications,  the  manner  in  which  the  completion-type 
test  items  were  constructed  was  roughly  similar  to  that 
used  in  Study  Two.  Minor  changes  were  made,  however,  in 
order  to  improve  the  objectivity  and  the  reliability  of  the 
test  writing  procedures.  Because  Bormuth's  work  on  rating 
scales  in  this  study  is  not  of  significance  to  this  re- 
searcher's investigation  of  cloze  criterion  scores  and 
information  gain,  the  details  of  that  portion  of  Study 
Three  will  not  be  treated.  Suffice  it  to  say,  however, 
that  each  student  also  was  required  to  rate  a reading 
passage  on  the  ten  preference  scales  mentioned  above 
(see  p . 60) . 

Using  the  above  materials,  Bormuth  constructed  his 
test  booklets.  Unlike  the  booklets  constructed  in  Study 
Two,  which  utilized  only  two  matched  passages  per  booklet, 
each  test  booklet  in  this  study  contained  four  passages, 
all  of  matched  difficulty.  Contained  in  order  within 
each  test  booklet  were  a cloze  test  made  from  one  of  the 
passages;  a completion  test  made  from  a second  passage 
(only  the  test,  however,  and  not  the  passage  was  in- 
cluded in  the  booklet) ; a third  passage  followed  by  the 
ten  preference  rating  scales;  and  finally,  a fourth  pas- 
sage followed  by  a 20-question,  completion-type  compre- 
hension test  based  on  information  found  in  that  fourth 
reading  passage. 

Tn  constructing  the  test  booklets,  the  four  matched 
passages  within  each  of  the  eight  difficulty  levels  were 
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first  coded  A through  D and  then  ordered  using  a Latin 
squares  design.  The  first  passage  in  each  order  was  admini- 
stered as  a cloze  test,  the  second  as  a prereading  test, 
the  third  was  rated  on  the  preference  scale,  and  the  fourth 
was  administered  as  a postreading  test  (see  Table  5). 

Five  versions  of  each  block  were  made,  one  for  each  form 
of  the  cloze  test,  which  resulted  in  20  test  booklets  at 
each  of  the  eight  difficulty  levels.  One  set  of  160  booklets 
constructed  at  each  of  the  ten  grade  levels  resulted  in 
the  production  of  1600  test  booklets.  According  to  Bormuth 
(1971),  each  student  "was  completely  nested  with  all  factors 
and  with  no  replications.  Thus,  there  were  160  students 
for  each  grade  level  and  200  students  for  each  difficulty 
level"  (p.  86). 


TABLE  5 

BORMUTH'S  LATIN  SQUARES  DESIGN  USED  IN  STUDY  THREE 


A B C D 
D A B C 
C D A B 


B C D A 


In  analyzing  the  findings  of  Study  Three,  Bormuth 
generally  used  the  same  procedures  that  had  been  employed 
in  Studies  One  and  Two,  i.et,  various  adjustments  were 
made  to  the  scores  and  the  data  were  subjected  to  re- 
gression analysis.  With  regard  to  the  relationship  between 
cloze  scores  and  information  gain,  as  reflected  by  a com- 
pletion-type comprehension  test,  Bormuth's  findings  in 
this  study  revealed  that  the  three  changes  in  his  design 
(see  p.  60)  resulted  in  regression  curves  that  conformed 
fairly  closely  to  the  shape  of  the  regression  obtained  in 
Study  One  (see  Figure  1).  Had  Bormuth's  test  writing 
procedures  used  in  Study  One  been  as  tightly  controlled 
as  those  used  in  Studies  Two  and  Three,  and  thereby  more 
fully  complied  with  his  instructional  conformity  and 
behavioral  consistency  assumptions,  the  shapes  of  the 
regression  curves  between  the  two  studies  would  undoubtedly 
have  been  even  more  similar,  if  not  identical.  The  fact 
that  the  regression  curves  between  Studies  One  and  Three 
were  very  similar,  and  could  have  been  made  even  more 
similar,  is  most  significant  because  it  confirms  that 
regression  curves  computed  over  many  passages  varying 
widely  with  regard  to  difficulty  are  nevertheless  similar 
to  those  computed  over  only  one  or  two  passages  of  equal 
difficulty.  This,  of  course,  substantiates  Bormuth's 
regression  identity  assumption,  which  says  that  the  regres- 
sion curves  must  not  change  regardless  of  the  number  of 
passages  used,  and  thereby  indicates  that  cloze  criterion 
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scores,  in  fact,  can  be  established.  Equally  signifi- 
cant is  the  fact  that  with  the  regression  identity  as- 
sumption having  been  confirmed,  subsequent  research  need 
not  be  conducted  over  many  passages  having  varied  diffi- 
culty levels.  Instead,  such  research  can  focus  on  only 
one  or  two  passages,  as  was  done  in  Bormuth's  Study  One, 
and  be  assured  that  any  findings  can  be  generalized  to 
all  passages  at  all  different  levels  of  difficulty. 

Nevertheless,  Study  Three  only  substantiated  one 
half  of  Bormuth's  regression  identity  assumption,  i.e., 

-i 

that  regression  curves  must  not  change  regardless  of  the 
passage  from  which  they  are  made.  As  Figure  1 shows, 
the  shape  and  slope  of  the  upper  end  of  the  regression 
curves  obtained  from  Study  Three  differed  for  each  grade 
level.  For  Bormuth,  whose  objective  was  to  establish 
maximum  comprehension  cloze  criterion  scores  gathered 
from  several  grade  levels  of  school  children,  many  of  whom 
had  not  fully  developed  the  reading  skill,  the  finding*, 
of  Study  Three  suggested  that  different  criterion  scores 
would  have  to  be  established  for  each  grade  level.  The 
fact  that  the  second  half  of  Bormuth's  regression  identity 
assumption  was  not  substantiated  in  Study  Three  is  not 
of  great  significance  for  this  project,  however,  which  is 
intended  to  establish  at  which  point  significant  compre- 
hension of  German  prose  reading  selections  begins  for 
adult  nonnative  readers  of  German.  The  reasons  for  its 
insignificance  are  twofold: 


■ 
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First,  whereas  Bormuth's  research  dealt  with  school 
children,  many  of  whom  had  not  fully  developed  the  skill 
of  reading,  this  research  project  is  limited  to  adults, 
who  although  reading  German  at  a number  of  different  skill 
levels  ranging  from  beginning  to  most  advanced,  are  all 
nevertheless  fully  developed  and  mentally  mature  readers. 
This  is  significant  because  in  analyzing  the  regression 
curves  of  students  in  grades  ten  through  12  in  Figure  1, 
it  is  clearly  evident  that  the  shapes  of  the  regressions 
of  these  mature  young  people  draw  much  closer  together. 

This  suggests,  of  course,  that  the  regression  identity 
assumption  is,  in  fact,  fully  met  for  adults  or  near 
adults  who  have  developed  their  reading  skills. 

Second,  whereas  Bormuth  is  directing  his  attention 
to  maximum  comprehension,  which  is  reflected  in  the  higher 
cloze  scores,  where  the  variance  between  grades  is  at  its 
maximum,  this  study  has  been  designed  to  examine  the 
lower  scores,  where  the  variance  between  grades  is  greatly 
reduced.  Therefore,  taking  into  consideration  the  fact 
that  this  study  is  using  only  adult  readers  and  is  examining 
their  scores  primarily  in  the  lower  half  of  the  spectrum, 
it  should  be  evident  that  only  the  first  half  of  Bormuth's 
regression  identity  assumption  need  be  considered  in  this 
research,  and  that  one  cloze  criterion  score  can  indeed 
be  established  for  all  adult  nonnativc  readers  of  German- - 
a cloze  score  that  is  valid  for  all  German  prose  reading 
selections  regardless  of  the  reader's  personal  rending 
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skill  or  the  difficulty  of  the  selection. 

In  examining  the  findings  of  Study  Three  with  regard 
to  establishing  that  point  at  which  comprehension  begins 
to  occur,  results  were  slightly  different,  but  probably 
more  accurate,  than  those  obtained  from  Study  One.  While 
Study  One  suggested  that  25  percent  on  the  cloze  test 
might  be  the  point  at  which  comprehension  began  to  occur, 
Study  Three,  which  was  much  more  comprehensive  and  opera- 
tionally replicable,  suggested  that  students  had  to  score 
at  least  35  percent  before  comprehension  began  to  occur. 

Although  this  author's  research  project  focuses  on 
nonnative  adult  readers  of  German  and  employs  a slightly 
different  design  incorporating  more  objective  test  writing 
procedures,  it  is  nevertheless  expected  that  a criterion 
score  somewhere  near  35  percent  will  also  be  established 
for  this  research  if,  in  fact,  the  three  research  questions 
stated  in  Chapter  I (see  p .<  28)  can  be  affirmatively 
answered. 

Summary 

The  purpose  of  Chapter  II  has  been  to  examine  that 
literature  related  either  directly  or  indirectly  to  the 
cloze  procedure  when  it  is  used  as  a.  measuring  device 
for  establishing  a meaningful  and  objective  cloze  cri- 
terion score  that  reflects  that  point  at  which  reading 
comprehension  begins  to  occur  for  nonnative  readers  of 
German  prose.  Tn  carrying  out  this  object  ice,  the  exami- 
nation of  literature  has  progressed  through  throe  general 
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stages : 

In  stage  one,  the  review  began  with  a very  general 
investigation  of  the  cloze  test  as  it  is  used  as  an  all- 
purpose measuring  device.  Because  so  much  research  has 
been  done  in  recent  years  involving  the  cloze  procedure 
as  a device  for  measuring  a variety  of  skills,  several 
comprehensive  reviews  focusing  on  the  entire  spectrum 
of  cloze  research  were  cited  (see  p.  32). 

In  stage  two  of  this  review  of  the  cloze  procedure, 
the  examination  was  narrowed  to  those  investigations 
involving  the  cloze  test  as  a measuring  device  to  be 
used  with  foreign  language  research.  During  this  phase 
of  the  review,  four  interesting  things  were  discovered: 
First,  especially  in  recent  years,  most  cloze  research 
carried  out  with  foreign  languages  has  been  intended  to 
measure  general  language  proficiency  rather  than  specific 
language  skills.  Nevertheless,  limited  work  in  foreign 
language  readability  and  reading  comprehension  work 
was  detected  and  cited.  Second,  most  foreign  language 
cloze  research  has  been  done  in  English  involving 
speakers  of  other  languages.  Third,  hardly  any  cloze 
research  has  been  conducted  involving  native  English 
speakers  using  a foreign  language.  Fourth,  although 
Anderson  (1976)  laid  some  important  groundwork  for 
foreign  language  cloze  research  and  reading  comprehension, 
no  research  was  detected  using  the  cloze  procedure  to 
assess  any  kind  of  reading  skills  on  nonnativc  speakers 
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• of  German. 

i.  In  stage  three,  the  final  area  of  cloze  research 

to  be  reviewed,  attention  was  focused  directly  on  the 
topic  of  this  research  investigation,  i.e.,  using  the 
cloze  procedure  as  a device  for  establishing  objective 
and  meaningful  reading  criterion  scores.  In  this  third 
area  of  investigation,  various  early  approaches  to  the 
problem  were  first  discussed.  Most  significant,  how- 
ever, was  the  research  reported  by  Bormuth  (1971)  in 
which  a series  of  five  studies  was  conducted  designed 
ultimately  to  establish  a rational  cloze  reading  criterion 
score  for  English-speaking  school  children.  Because  of 
their  direct  bearing  on  this  author's  research  project, 
only  the  first  three  of  Bormuth's  five  research  projects 
were  dealt  with  in  this  review.  However,  because  these 
three  studies  form  the  backbone  of  this  research  project, 

* they  were  dealt  with  in  considerable  detail. 


CHAPTER  III 


PROCEDURES 

Chapter  III  is  divided  into  five  major  sections: 

1)  a general  discussion  of  the  design  employed  to  gather 
the  data;  2)  an  in-depth  discussion  and  description  of  the 
rules  developed  and  applied  in  order  to  ensure  that  both 
testing  instruments  would  be  operationally  replicable; 

3)  an  examination  of  all  the  details  involved;  4)  an  ex- 
planation of  the  criteria  established  for  scoring  the 
findings;  and  5)  a brief  discussion  of  the  statistical 
procedures  used  to  analyze  the  data. 

Design 

The  purpose  of  this  research  study  is  to  establish 
an  objective  and  meaningful  cloze  criterion  score  that 
reflects  that  cloze  level  at  which  college- level , non- 
native readers  of  German  prose  begin  to  glean  substantial 
information  as  a result  of  reading.  The  value  of  estab- 
lishing such  an  objective  cloze  criterion  score,  of  course, 
is  that  it  thereby  enables  teachers  of  German  to  rationally 
and  ohjectively  identify  reading  selections  that  are  of 
an  appropriate  reading  difficulty  level  for  their  students. 
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The  objective  is  to  be  accomplished  by  comparing  and 
then  establishing  the  relationship  of  scores  carefully 
matched,  college- level , nonnative  speakers  of  German  achieve 
on  a cloze  test  with  the  scores  other  matched,  college- 
level,  nonnative  speakers  of  German  achieve  on  a specially 
prepared,  operationally  replicable,  short-answer  completion 
test  constructed  over  the  same  piece  of  German  prose. 

Because  neither  test  was,  nor  could  be,  manipulated  in 
any  way  by  the  researcher,  this  investigation  is  of  neces- 
sity descriptive  research  and  according  to  Kerlinger 
(1973),  is  necessarily  ex  post  facto  in  nature. 

Design  Components 

The  specific  design  employed  in  this  investigation 
was  inspired  by  Bormuth‘s  series  of  research  studies 
conducted  in  1971  in  English.  The  simplicity  of  the  design 
is  directly  attributable  to  Bormuth,  who, as  a result  of 
findings  achieved  primarily  in  Study  Three  (see  p.  59), 
was  able  to  substantiate  the  regression  identity  assump- 
tion that  regression  curves  computed  over  many  passages 
of  varying  difficulty  were  nevertheless  similar  to  those 
computed  over  only  one  or  two  passages  of  equal  diffi- 
culty. As  a result,  the  design  used  in  this  study  could 
be  simplified  to  include  only  two  reading  passages,  one 
to  match  the  students  according  to  reading  ability,  and  one 
to  gather  the  actual  data  for  analysis,  and  nevertheless 
still  be  assured  that  findings  could  be  generalized  to 
ill  passages  at  any  difficulty  level,  a requirement 
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necessary  for  establishing  meaningful  and  generalizable 
cloze  criterion  scores. 

In  collecting  the  data  used  in  this  research  project, 
the  following  steps  were  accomplished:  First,  because 
researchers  (Bormuth,  1962  and  Fletcher,  1955)  had  demon- 
strated the  cloze  procedure  to  be  both  a valid  and  reli- 
able measure  of  reading  comprehension  ability,  the 
students  selected  to  participate  in  the  study  were  initi- 
ally given  a cloze  test  constructed  from  a reading  selection 
of  German  prose  in  order  to  form  pairs  of  readers  matched 
according  to  reading  ability. 

Second,  a second  selection  of  German  prose  was  prepared 
in  two  different  versions  for  testing:  1)  Three  different 
forms  of  the  cloze  test  were  constructed  in  which  different 
combinations  of  every  eighth  word  were  deleted.  For  example, 
on  one  cloze  form,  words  2,  10,  18,  etc. were  deleted;  on 
a second  form,  words  4,  12,  20,  etc.  were  deleted;  and  on 
the  final  cloze  form,  words  5,  13,  21,  etc.  were  deleted. 
Multiple  forms  of  the  cloze  test  were  used  to  control  for 
sampling  error  and  ensure  that  the  actual  difficulty  of 
the  passage  was  being  sampled  and  tested.  2)  A specially 
prepared,  operationally  replicable,  short-answer  comple- 
tion test  was  constructed  using  guidelines  outline  in  this 
chapter  (see  p . 76) . 

Third,  one  member  from  each  matched  pair  was  randomly 
selected  and  administered  the  specially  prepared  completion 
test,  while  the  other  matched-pair  member  was  administered 
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one  of  the  three  forms  (randomly  chosen)  of  the  cloze 
test  (see  Figure  2).  The  scores  achieved  from  these  two 
test  types  were  then  compared  in  order  to  determine  what 
kind  of  relationship  existed,  if  any. 

Instrument  Development 

In  order  to  establish  meaningful  and  objective  cloze 
reading  criterion  scores  based  on  information  gain,  it  is 
necessary  to  compare  the  relationship  of  cloze  test  scores 
and  objective  test  scores.  For  this  comparison  to  be 
valid  and  thereby  general i zable  to  all  German  prose  reading 
selections,  however,  it  is  essential  that  both  Bormuth's 
instructional  conformity  and  behavioral  consistency  as- 
sumptions be  met.  To  meet  these  two  assumptions,  both 
the  cloze  test  and  the  objective  test  must  be  prepared  in 
an  operationally  replicable  manner.  By  their  very  nature, 
cloze  tests  are  replicable  because  they  are  prepared  by 
systematically  deleting  words  according  to  a set  of  pre- 
determined rules.  Objective  tests,  however,  are  normally 
generated  in  a very  subjective  manner,  being  greatly 
influenced  by  the  individual  biases  of  each  test  writer. 

An  integral  and  essential  part  of  this  research  project, 
therefore,  was  to  develop  an  objective  test  writing 
algorithm  in  which  a set  of  rules  was  generated  for  con- 
structing an  operationally  replicable,  short- answer , 
completion- type  objective  test  in  German.  In  this  chapter 
rules  used  for  generating  both  the  objective  test  and  the 
cloze  test  will  be  set  forth. 
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FIGURE  2.  Research  design  used  to  gather  data  for  analysis. 
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The  Objective  Test 

In  developing  the  following  algorithm  for  writing 
replicable,  completion- type  objective  tests,  initial 
inspiration  was  drawn  from  Bormuth  (1970),  who  developed 
test  writing  procedures  for  English  texts  based  on  genera- 
tive transformational  grammar.  The  actual  test-writing 
algorithm  used  in  this  study,  however,  was  patterned  after 
ideas  suggested  by  Finn  (1973),  who  was  inspired  by  research 
conducted  by  Stockwell,  Schachter,  and  Partee  (1968),  in 
which  the  theories  of  both  generative  transformational 
grammar  and  case  grammar  were  integrated  to  develop  a very 
detailed  grammar  of  English. 

The  algorithm  used  in  this  study  contains  33  test- 
writing rules  divided  into  four  general  sections:  1) 
rules  for  sentence  selection,  2)  rules  for  sentence  pre- 
paration, 3)  rules  for  sentence  analysis,  and  4)  rules 
for  question  preparation.  In  presenting  the  rules  of 
this  test-writing  algorithm,  each  rule  will  be  stated  and 
then  followed  by  an  explanation  and/or  example,  where 
necessary. 

Rules  for  Sentence  Selection 

A key  first  step  in  generating  an  operationally  rep- 
licable test-writing  algorithm  is  to  determine  which  sen- 
tences within  the  chosen  reading  selection  contain  infor- 
mation or  ideas  of  such  significance  to  the  passage  that 
their  content  should  be  considered  for  inclusion  as  one 
of  the  test  questions.  To  make  such  a determination,  the 
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following  rules  must  be  applied: 

Rule  1.  Determine  each  word's  Text  Frequency  (TF) 
in  the  reading  selection  by  counting  how  often  it 
occurs  in  the  passage. 

Explanation  and  F.xamplc.  If  there  were  an  interest 
in  writing  a completion  test  over  a 500-word  German  sele- 
ction about  science.  Rule  1 dictates  that  one  simply  would 
first  count  how  often  every  word  in  the  passage  occurred. 
The  frequently  used  definite  article  "der,"  for  example, 
might  occur  25  times  in  the  selection  and  therefore  have 
a Text  Frequency  of  25,  while  the  noun  "Atom"  might  occur 
only  five  times  and  therefore  have  a Text  Frequency  of 
only  five. 

Rule  2.  Compute  an  Interim  Value  (IV)  for  each  word 
in  the  passage  by  dividing  each  word's  Text  Frequency 
by  the  total  number  of  words  (TW)  contained  within 
the  reading  selection. 

Explanation.  With  "der"  occurring  25  times  in  the 
500-word  reading  selection  about  science,  it  would  be 
given  an  Interim  Value  of  .05,  while  "Atom,"  occurring 
only  five  times,  would  be  assigned  an  Interim  Value  of  .01. 
Example. 

IV  for  "der"  - TF  = = .05 

IV  for  "Atom"  - iJJJ  - - .01 

Rule  3 . Determine  the  Meier  Frequency  (MF)  of  all 
words , i . c . , how  frequently  each  word  occurs  in  a 
count  of  nearly  11  million  (10,910,777)  running 
German  words,  by  consulting  Meier's  Deutsche  Sprach- 
stat i st i k (1967). 
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Explanation . Utilizing  the  definitive  work  llHuf igkeits 
wbrterbuch  der  deutschen  Sprache  compiled  at  the  end  of  the 
19th  Century  by  F.  IV.  Kaeding,  Meier  has  alphabetically 
listed  every  form  of  every  German  word  that  appears  ten  or 
more  times  in  Kaeding's  work.  Meier's  work  consists  of 
41,083  separate  word  forms  and  lists  how  frequently  each 


occurs  in  Kaeding's  count.  Meier's  list  constitutes  96.3 

percent  of  all  words  found  in  running  German  texts. 

Example . "Der"  has  a Meier  Frequency  of  341,522. 

"Atom,"  on  the  other  had,  has  a Meier  Frequency  of  only  14. 

Rule  4.  In  order  to  determine  each  individual  word's 
importance  to  passage  understanding,  compute  the 
Significance  Coefficient  (SC)  of  all  words  in  the  passage 
by  dividing  each  word's  Interim  Value  by  its  Meier 
Frequency.  If  a word  is  not  listed  by  Meier,  auto- 
matically assign  that  word  a Meier  Frequency  value 
of  nine  (9)  . 

Explanation . The  intent  of  Rule  4 is  to  identify 
those  words  truly  important  to  passage  understanding  and 
to  prevent  words  such  as  articles  and  prepositions,  which 
always  occur  frequently,  from  being  selected  as  signifi- 
cant to  passage  understanding  just  because  they  occur 
frequently. 


Example . 
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Explanation . That  word  with  the  highest  Significance 
Coefficient  will  be  listed  as  word  one;  the  word  with  the 
next  highest  Significance  Coefficient  will  be  listed  as 
word  two,  etc.,  until  the  SO  most  significant  words  have 
been  identified.  In  those  cases  where  there  are  two  or 
more  words  with  the  same  Significance  Coefficient,  they 
will  be  arranged  alphabetically  and  numbered  consecutively. 
Example . See  Appendix  C,  p.  173. 

Rule  6.  Locate  and  list  numerically  those  sentences 
in  the  reading  selection  in  which  each  of  the  50 
most  significant  words  occurs.  These  are  the  sentences 
that  will  be  used  in  generating  questions. 

Explanation  and  Example.  If  "Atom"  had  the  highest 

Significance  Coefficient  in  a 500-word  passage  on  science 

and  occurred  five  times  in  the  passage,  question  one  would 

be  derived  from  the  first  sentence  in  the  selection  in 

which  "Atom"  occurred.  Question  two  would  be  derived  from 

the  second  sentence  in  the  passage  in  which  "Atom"  occurred, 

etc.,  until  all  five  sentences  containing  the  word  "Atom" 

had  been  utilized  in  generating  questions. 

Rules  for  Sentence  Preparation 

After  identifying  those  sentences  within  a reading 

selection  that  contain  words  key  to  passage  understanding, 

steps  must  be  taken  to  modify  these  sentences  into  their 

basic  semantic  components  in  preparation  for  structurally 

analyzing  them.  To  modify  selected  sentences  into  their 

basic  semantic  components,  the  following  rules  must  be 

applied: 
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Rule  7.  Replace  all  pronouns  with  the  appropriate 
noun . 

Example . "Dort  lebte  er  auf  einer  Oase  ganz  wie  in  J 

England."  becomes  "Port  lebte  der  Engl  Under  auf  einer  Oase 
ganz  wie  in  England." 

Rule  8.  Replace  adverbs  such  as  "dann,"  "spHter," 

"dort , " etc.  with  the  appropriate  phrase  or  clause.  ‘ 

Example . "Dort  lebte  der  Englander  auf  einer  Oase 
ganz  wie  in  England."  becomes 'In  der  WUste  lebte  der 

t 

1 

Englander  auf  einer  Oase  ganz  wie  in  England." 

Rule  9.  Treat  all  compound  sentences  joined  by  "und" 

as  separate  sentences  ignoring  the  conjunction.  < 

Example . "Der  Englander  badete  fleissig  und  trank 

viel  Tee."  becomes  "Der  Englander  badete  fleissig."  "Der 

Englander  trankviel  Tee." 

Rule  10.  If  an  adverb/adverbial  (ADV)  is  present  in 
a sentence  analyzed  as  more  than  one  sentence,  repeat 
the  ADV  in  all  sentences. 

Example . "Jeden  Tag  badete  der  Englander  fleissig 

und  trank  viel  Tee."  becomes  "Jeden  Tag  badete  der  EnglHnder 

fleissig."  and  "Jeden  Tag  trank  der  Englander  viel  Tee." 

Rule  11.  Rewrite  sentences  with  elliptical  constructions 
restoring  the  word  or  words  that  complete  the  con- 
struction . 

Example . "Guten  Tag."  becomes  "Ich  wdnsche  Ihnen 
einen  guten  Tag." 

Rules  for  Sentence  Analysis 

Having  selected  and  modified  into  their  basic  semantic 
components  those  sentences  that  contain  words  and  infor- 
mation key  to  passage  understanding,  sentences  arc  now 
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ready  for  structural  analysis.  In  structurally  analyzing 

sentences,  the  following  rules  must  be  followed:* 

Rule  12.  Divide  each  prepared  sentence  (S)  into 
modal ity  (M)  and  proposition  (P)  . 

Example . 


S 


M P 


Rule  13.  Under  modality,  place  negative  elements 
(nicht) ; adverbs  (sofort,  ganz,  schnell,  etc.);  de- 
pendent clauses  functioning  as  sentence  adverbials 

(weil  der  Mann  sehr  krank  ist etc . ) ; and  other 

elements  understood  as  modalities  on  the  sentences 
as  a whole  (tense;  voice,  other  than  active;  use  of 
modal  verbs) . 

Example . Following  Rule  13,  in  the  sentence  "Jeden 
Tag  badete  der  Englander  fleissig.”,  modality  would  be 
constructed  in  the  following  manner: 


Rule  14.  Under  proposition,  place  the  verb  (V)  plus 
phrases  having  case  relationships  (C)  to  the  verb. 


^Examples  provided  with  the  following  rules  are,  of 
necessity,  very  elementary.  For  more  detailed  constructions 
and  analyses  sec  Appendix  I),  p.  1 7 8. 
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Example . Following  Rule  14,  in  the  sentence  "Jeden 
Tag  badete  der  Englander  fleissig.",  the  following  ele- 
ments would  be  placed  under  proposition: 


S 


baden  der  Englander  jeden  Tag 


Rule  15.  Do  not  treat  "sein"  as  a verb.  If  one  of 
its  conjugated  forms  is  followed  by  a predicate  ad- 
jective, enter  the  predicate  adjective  as  the  verb. 
Otherwise,  assign  any  form  of  "sein"  the  value  of 
zero  (0) . 

Example . Following  Rule  15,  in  the  sentence  "Der 
Englander  war  in  England.",  the  verb  form  "war"  would  be 
given  the  value  of  zero: 


S 


Rule  16 . Analyze  all  case  relationships  (C)  as  Prep- 
' ositions  (K),  which  indicate  the  relationship  of  the 

phrase  to  the  verb,  and  noun  phrases  (NP) . K may 
be,  and  often  is,  zero  (0). 

Example . Following  Rule  16,  in  the  sentence  "Der 


EnelMiulcr  wohnt  in  Enuland."  the  case  relationships  fCl 


under  the  proposition  (P)  would  be  further  broken  down  as 
follows : 
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Rule  17.  Construe  all  relative  clauses  (regular  and 
reduced)  as  sentence  (S)--right  sister  to  the  noun 
phrase  (NP)  and  under  the  noun  Phrase  (NP)  . 


Example . Following  Rule  17,  in  the  sentence  "Der 
grosse  Englander  wohnt  in  England.",  the  reduced  relative 


clause  "Der  grosse  Englander  would  be  treated  as  follows: 


S 


r 
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Rule  18.  When  a phrase  appears  to  be  analyzable  either 
as  a reduced  relative  clause  under  NP  or  as  an  adver- 
bial under  C,  as  in  "Der  EnglHnder  studierte  das 
Kamel  in  der  Wllste.",  analyze  the  phrase  as  C. 

Example . Following  Rule  18,  in  the  sentence  "Der 

Engl'lnder  studierte  das  Kamel  in  der  Wllste.",  the  phrase 

"in  der  Wllste"would  be  treated  as  follows: 


Rule  19.  Do  not  analyze  ordinals  (der  erste,  der 
n^chste , der  letzte,  etc.)  or  quantifiers  (einige, 
viele  mehrere,  etc.)  as  S under  NP.  Instead,  the 
head  noun  and  these  modifiers  arc  to  be  treated  as 
an  inseparable  unit  under  C. 

Example . Following  Rule  19,  the  sentence  "Der  letzte 
Mann  wohnte  in  England."  would  be  analyzed  as  follows: 


Rule  20.  Treat  all  problem  modifiers  not  dealt  with 
in  Rules  17,  18,  and  19  as  part  of  the  head  word. 

Rule  21.  Allow  only  S to  be  the  daughter  of  NP. 

Explanat ion . Rule  21  compliments  and  supports  Rule 

17  by  allowing  only  for  an  S to  be  directly  attached  to  an  NP. 

Example . Following  Rule  21 , in  the  sentence  "Der 

grosse  Englander  wohnt  in  England.",  the  reduced  relative 

clause  "der  grosse  Englander"  must  be  represented  as  an  S 

under  NP  as  follows: 


S 


Rule  22.  Analyze  conjoined  constituents  as  compound 
constituents  under  C. 

Example . Following  Rule  22,  in  the  sentence  "Der 


Englander  spielte  Tennis  und  Cricket.",  the  conjoined 


constituents  "Tennis  und  Cricket"  must  be  represented  as 
follows : 


Rule  23.  Analyze  genitive  constituents  as  extensions 
of  the  C to  which  they  refer. 

Example . Following  Rule  23,  in  the  sentence  "Der 
EnglHnder  stellte  eine  Beschreibung  dcs  Kamels  zusammen.", 
the  genitive  "des  Kamels"  must  be  represented  as  follows: 


Rules  for  Question  Prcpnrntio n 

Having  selected,  modi  Tied,  and  structurally  analyzed 
those  sentences  containing  words  and  information  key  to 


passage  understanding,  steps  now  may  be  taken  to  generate 
actual  test  questions.  In  transposing  sentences  into 
questions,  the  following  rules  must  be  applied: 
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Rule  24.  In  selecting  that  sentence  constituent 
to  be  replaced  with  an  interrogative  form,  consider 
only  case  constituents  labeled  Cj...C  and  sentence 
constituents  labeled  Sj...Sn. 

Example . Following  Rule  24,  in  the  sentence  "Der 
grosse  EnglSnder  wohnt  in  England.",  only  constituents 
Cj,  C2,  and  Sj  may  be  considered  for  replacement  with  an 
interrogative  form: 


S 


Rule  25.  Determine  the  constituent  to  be  deleted 
By  means  of  random  selection. 

Rule  26.  If  either  a C or  S constituent,  which  has 
subordinate  S(j  constituents,11  i s selected  for  replacement, 
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delete  the  subordinate  constituents  as 
Explanation.  Using  the  example  presented 
24,  Rule  26  directs  that  should  constituent 
selected  for  replacement  with  an  interrogative 


well . 

under  Rule 
be  randomly 
form , 


constituent  would  be  deleted  as  well  because  it  is 
subordinate  to  Cj.  If  Sj  should  be  randomly  chosen,  however, 
constituent  Cj  would  remain  because  it  is  superordinate 
to  Sj 

Rule  27.  In  each  analyzed  sentence,  replace  the 
randomly  deleted  constituent  with  one  of  the  fol- 
lowing interrogative  forms:  "wann,"  "warum,"  "was," 

"was  ftir,"  "was  + tun,"  "welcher,"  "wer,"  "wie," 

"wie  lange,"  "wo,"  "woher,"  "wohin,"  "wo(r)  + Prep- 
osition." 

Example.  Using  the  sentence  presented  under  Rule 
24,  had  C2  (in  England)  been  randomly  selected  for  inter- 
rogative replacement,  semantic  meaning  would  require  that 
the  interrogative  "wo"  be  selected  from  among  the  choices 
listed  in  Rule  27  to  generate  the  question. 

Rule  28.  If  in  generating  a question  a C is  randomly 
selected  for  interrogative  replacement,  piace  the 
appropriate  interrogative  to  the  front  of  the  sent- 
ence and  follow  German  inverted  word-order  rules. 

Example . Using  the  sentence  presented  under  Rule 
24,  had  C2  (in  England)  been  randomly  selected  for  inter- 
rogative replacement,  the  question  would  read  as  follows: 

"Wo  wohnt  der  grosse  Englander?" 


Rule  29.  If  in  generating  a question  an  S is  randomly 
selected  for  interrogative  replacement,  follow  Rule 
28  unless  the  Sn  to  be  deleted  is  subordinate  to  a 
Cn  containing  a K with  a value  other  than  zero  (0). 

In  such  instances,  begin  the  question  witli  the  K 
followed  by  the  semantically  appropriate  interrogative. 
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Example . Following  Rule  29,  if  were  randomly 
selected  for  interrogative  replacement  in  the  sentence 
"Der  Englander  wohnt  in  einem  alten  Haus.",  represented 
below,  the  resultant  question  would  read  as  follows:  "In 
was  fUr  einem  llaus  wohnt  der  Englander?" 


Rule  30.  Although  the  indefinite  pronoun  "man" 
and  its  declined  forms  "einen"  and  "einem"  may  serve 
in  a Cn  capacity,  do  not  consider  them  for  deletion. 


Rule  31.  Should  selection  procedures  involving  two 
adjacent  questions  result  in  the  same  answer  for 
both  questions,  randomly  select  another  constituent 
for  deletion  from  the  latter  question. 


Rul c 32 . Should  any  constituent  chosen  for  deletion 
prove  to  be  stylistically  unacceptable,  eliminate 
that  constituent  from  consideration  and  randomly 
select  from  the  remaining  constituents. 
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Rule  33.  After  a question  has  been  generated  from 
a sentence,  do  not  reuse  the  sentence  a second  time 
even  though  it  may  contain  one  or  more  additional 
significant  words. 

The  Cloze  Test 

Unlike  the  objective  test,  which  is  most  difficult 
to  make  operationally  replicable,  the  cloze  test  generally 
presents  no  difficulties.  In  fact,  according  to  Rankin 
(1965),  "one  of  the  most  characteristic  features  of  the 
cloze  procedure  is  the  use  of  a completely  mechanical 
system  for  choosing  words  to  be  deleted"  (p.  144).  Being 
"completely  mechanical,"  of  course,  implies  operational 
replicability.  Therefore,  in  constructing  a cloze  test, 
the  primary  task  is  not  in  ensuring  the  cloze  test  will  be 
operationally  replicable,  but  rather  in  determining  what 
form  of  replicability  the  test  will  take,  i.e.,  what  rules 
will  be  followed  in  mechanically  deleting  words. 

Two  Types  of  Cloze 

In  constructing  cloze  tests,  the  two  most  commonly 
used  systems  for  choosing  words  to  be  deleted  are  random 
deletion  and  an  every  nth  deletion.  Because  the  every 
nth  deletion  is  easier  to  construct,  most  researchers  have 
preferred  this  method.  In  selecting  an  every  nth  deletion, 
however,  the  question  is  immediately  raised  concerning 
what  deletion  rate  to  use.  I!or  example,  in  addition  to 
experimenting  with  random  deletion,  Taylor  (1953)  also 
explored  deleting  every  fifth,  seventh,  and  tenth  word. 
According  to  Anderson  (1976),  Anderson  (1965)  deleted 
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every  eighth  word.  Weaver  (1961)  every  tenth  word,  and 
Greene  (1965)  every  twelfth  word.  The  best  controlled 
research  was  that  of  MacGinitie  (1961)  who  compared  the 
effect  of  deleting  every  third,  sixth,  twelfth,  and  twenty- 
fourth  word  and  concluded  that  "additional  uninterrupted 
context  beyond  five  words  did  not  help  in  the  restoration 
of  the  missing  words"  (p.  127).  Based  on  MacGinitie's 
work,  then,  suggesting  that  five-word  deletions  resulted 
in  statistical  independence,  most  research  involving  the 
cloze  procedure,  including  the  work  of  Bormuth,  has  em- 
ployed the  every  fifth-word  deletion  pattern.  As  Rankin 
(1965)  points  out,  however: 

It  does  not  follow,  however,  that  this  pattern 
(every  fifth-word  deletions)  is  suitable  for  all 
materials,  age  groups,  and  purposes  for  which  the 
cloze  procedure  may  be  used.  More  work  is  needed 
on  this  question  (p.  145). 

As  has  already  been  pointed  out  in  Chapter  II,  almost 
all  cloze  research  has  been  conducted  in  English  using 
native  speakers  of  that  language.  However,  according  to 
Oiler  (1973),  "there  is  little  if  any  reason  to  assume 
that  conclusions  from  research  with  native  speakers  can 
validly  be  generalized  to  the  case  of  non-native  speakers" 
(p.  107).  Therefore,  although  it  has  been  determined  with 
native  speakers  that  deleting  more  than  one  out  of  every 
five  words  in  a cloze  test  creates  a test  so  difficult 
that  discriminatory  power  is  lost,  and  to  delete  less 
frequently  than  one  in  12  docs  not  substantially  change 
the  quality  of  the  test,  such  findings  do  not  suggest  how 
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many  words  should  be  between  deletions  when  testing  non- 
native speakers  of  German.  All  that  is  suggested  is  that 
both  for  native  and  nonnative  speakers,  five-word  deletions 
will  be  more  difficult  than  ten-word  deletions.  To 
be  any  more  specific  would  result  in  subjectivity  because 
according  to  Oiler  (1973),  "the  parameter  of  distance 
between  blanks  has  not  been  investigated  systematically 
with  non-native  speakers"  (p.  107). 

Justification  for  an  Eight-Word  Deletion 

Based  on  a lack  of  specific  research  providing  dele- 
tion guidance  for  nonnative  speakers  of  German,  an  eight- 
word  cloze  deletion  system  was  selected  in  this  research 
for  two  reasons:  1)  Because  students  who  possessed  varying 
degrees  of  German  language  reading  proficiency,  ranging 
from  very  elementary  to  near  native,  were  needed  in  order 
to  determine  at  which  point  on  the  cloze  test  signifi- 
cant comprehension  began  to  occur,  it  was  felt  that  the 
difficulty  index  of  five-word  deletions  ultimately  might 
be  so  high  as  to  invalidate  inordinately  large  proportions 
of  the  sample  and  therefore  adversely  affect  the  findings. 
Using  eight-word  deletions,  on  the  other  hand,  would  probably 
significantly  reduce  the  difficulty  index  and  likely 
result  in  more  usable  data  spread  over  a broader  range 
of  readers.  2)  A pilot  study  involving  near-native  readers 
revealed  that  a five-word  deletion  system  applied  to  the 
passages  selected  for  this  research  was  very  time  con- 
suming for  the  students  and  required  approximately 
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30  minutes  to  complete.  Because  all  data  had  to  he  col- 
lected in  a 50-minute  class  period,  and  all  readers  at  all 
levels  needed  ample  time  to  complete  the  selections,  it 
was  decided  that  eight-word  deletions  would  allow  less- 
skilled  readers  additional  time  to  complete  the  cloze 
test.  Therefore,  the  cloze  tests  used  in  this  research 
followed  every  eighth-word  deletion  patterns. 

Study  Details 

In  discussing  the  details  of  this  project,  three 
areas  will  be  examined:  the  subjects  utilized,  the  pas- 
sages employed,  and  the  testing  sequence  followed. 

The  Subjects 

The  sample  used  in  this  research  was  drawn  from  a 
target  population  of  college-aged,  nonnative  students  of 
German  who  had  completed  a minimum  of  two  college  quarters 
of  formal  German  instruction  at  one  of  two  central  Ohio 
institutions  (The  Ohio  State  University  and  Otterbein 
College) . Because  the  goal  was  to  establish  a cloze 
criterion  score  by  determining  at  which  percentage  point 
on  the  cloze  test  nonnative  readers  of  German  began  to 
glean  substantial  information,  it  was  essential  that  the 
sample  be  relatively  large  and  contain  proportionate  numbers 
of  students  whose  German  reading  ability  ranged  from  very 
elementary  through  to  near  native.  Therefore,  315  students 
enrolled  in  27  classes  differing  in  difficulty  from  third 
quarter  first  year  through  to  the  graduate  level  were 
selected.  Due  to  absences,  students  enrolled  in  two  or 
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more  classes,  and  students  desiring  not  to  participate 

in  the  project,  210  students  (108  males  and  102  females)  j 

were  ultimately  utilized.  A completely  random  selection  1 

j 

of  the  sample  from  the  population  was  not  accomplished  for 

1 

two  reasons:  1)  only  a limited  number  of  nonnative  stu- 
dents was  available,  and  2)  a relatively  large  sample  of  J 

, i I 

at  least  200  students  was  desired.  Nevertheless,  random 
assignment  of  students  to  treatment  was  accomplished. 

Because  only  partial  randomization  was  accomplished,  the  1 

findings  of  this  study  can  only  be  generalized  to  the  ! 

larger  population  of  all  col lege- aged , nonnative  students  \ 

of  German  by  that  degree  to  which  the  sample  used  in  this  < 

study  is  representative  of  that  population.  There  is  no 
apparent  reason  to  believe,  however,  that  the  students 
involved  in  this  study  were  significantly  unlike  those 
college-aged,  nonnative  students  enrolled  at  other  insti- 
tutions throughout  the  country,  which  suggests  that  the 
findings  obtained  in  this  project  are  meaningful  and 
generally  can  be  applied  to  the  larger  population. 

The  Passages 

A great  deal  of  forethought  was  carried  out  in  select- 
ing the  two  passages  used  in  this  research.  At  least  three 
factors  needed  careful  consideration:  First,  in  order  to 
ensure  that  the  difficulty  level  of  the  reading  selections 
would  be  accurately  assessed,  it  was  necessary  to  select 
passages  of  such  a length  that  cloze  tests  containing  at 
least  50  blanks  could  be  constructed.  This  determination 
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was  made  based  on  two  inputs:  1)  a comment  by  Taylor 
(1956),  who  indicates  that  "a  series  of  about  50  blanks 
is  roughly  sufficient  to  allow  the  chances  of  mechanically 
selecting  easy  or  hard  words  to  cancel  out  and  yield  a 
stable  score  of  the  difficulty  of  a passage"  (p.  48); 
and  2)  research  conducted  by  Bormuth  (1964),  who  investi- 
gated the  variability  of  passage  difficulty  at  different 
test  lengths,  found  that  differences  in  difficulty  dimini- 
shed as  test  length  increased,  and  recommended  that  cloze 
tests  contain  at  least  50  blanks. 

Second,  in  addition  to  having  reading  selections 
from  which  cloze  tests  of  at  least  50  blanks  could  be 
constructed,  finding  passages  of  an  appropriate  difficulty 
level  was  critical.  Because  it  was  essential  for  the  sample 
to  contain  students  possessing  reading  skills  ranging 
from  very  elementary  to  near  native  in  order  to  determine 
at  which  cloze  point  readers  began  to  gain  substantial 
information,  it  was  also  necessary  to  choose  reading  sele- 
ctions that  were  neither  too  easy  nor  too  difficult.  For 
example,  were  the  passages  to  prove  to  be  either  too  easy 
or  too  hard,  all  readers  in  the  sample  would  achieve  either 
very  high  or  very  low  scores.  However,  without  a wide 
range  of  scores,  it  would  be  most  difficult  to  determine 
a cloze  criterion  score  reflecting  that  cloze  point  where 
information  began  to  be  gleaned. 
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Third,  it  was  decided  to  use  passages  based  on 
fiction  rather  than  fact  in  order  to  eliminate  the  need 
for  administering  the  objective  test  twice  during  two  sep- 
arate testing  sessions,  once  as  a prereading  test  and 
once  as  a postreading  test  to  determine  how  much  prior 
knowledge  each  student  taking  the  objective  test  had  of  the 
material  in  the  reading  passage.  Using  a passage  of  un- 
familiar fiction  guaranteed  that  no  one  would  possess  any 
prior  knowledge  of  the  information  in  the  reading  selec- 
tions and  thereby  would  also  eliminate  the  need  for  a 
pretest  administered  during  an  extra  testing  session. 

Taking  the  above  three  factors  into  consideration 
(length  of  passage,  difficulty  level,  and  subject  matter), 
two  reading  passages  were  selected,  one  from  which  a cloze 
test  was  made  to  match  students  according  to  reading  abil- 
ity, and  one  from  which  both  a cloze  test  (three  different 
forms)  and  an  operationally  replicable,  short-answer, 
completion- type  objective  test  were  constructed  for  use 
in  the  data  analysis.  Because  of  the  extreme  importance 
of  difficulty  level,  prior  to  final  passage  selection 
all  potential  passages  were  reviewed  by  an  experienced, 
widely  published  native- German  writer  of  graded  German 
reading  selections.  Based  on  author  evaluation  of  the 
difficulty  level  of  the  work  and  subsequent  agreement 
and  approval  by  the  experienced  German  writer,  "Die 
Riesen  und  die  Zwerge,"  a 493-word  fairy  talc  was  se- 
lected as  a basis  for  constructing  a cloze  test  to  match 
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students  according  to  reading  ability,  and  "Das  Kfmel," 
a 572-word  fictitious  story  was  chosen  from  which  to  generate 
the  cloze  and  objective  tests  for  use  in  the  actual  data 
analysis.  Both  stories  came  from  a German  graded  reader 
entitled  GlUck  Auf  by  MUller  and  Wenckebach  (1901).  From 
"Die  Riesen  und  die  Zwerge"  was  constructed  a 58-blank 
cloze  test.  From  "Das  Kamel"  were  constructed  three  dif- 
ferent forms  of  a 69-blank  cloze  test  and  a 38-question, 
completion- type  objective  test  that  was  operationally 
generated  (see  Appendix  B). 

Testing 

In  order  to  gather  the  necessary  data  for  analysis, 
two  testing  sessions  with  all  participants  were  required, 
one  for  matching  students  according  to  reading  ability, 
and  one  for  collecting  the  data  to  be  analyzed.  In  prep- 
aration, permission  was  received  from  18  university  teachers 
(three  teaching  assistants,  three  instructors,  and  12 
professors)  to  visit  their  27  respective  German  classes 
during  two  separate  class  periods. 

In  the  first  testing  session,  approximately  40  minutes 
of  class  time  were  required.  In  order  to  motivate  all 
participants,  the  importance  of  reading  appropriately 
difficult  reading  selections  was  first  emphasized.  Brief 
written  instructions  for  taking  the  cloze  test  were  then 
reviewed,  and  the  test  was  administered  (see  lest  I in 
Appendix  B) . In  matching  pairs  of  students  according  to 
reading  ability,  all  scores  were  first  converted  to  a 
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percentage,  ranked  according  to  size,  and  then  matched 
in  order,  a pair  at  a time.  In  only  seven  of  105  matched 
pairs  did  the  cloze  scores  differ  by  more  than  .9  percentage 
points.  In  no  instance  was  the  difference  greater  than 
3.5  percentage  points.  As  a general  indication  that  a 
broad  range  of  reading  skills  was  contained  within  the 
sample,  and  that  the  passage  "Die  Riesen  und  die  Zwerge" 
had  been  of  an  appropriate  difficulty  level,  the  high 
score  achieved  was  97.4  percent,  the  low  score  was  12.1 
percent,  and  the  mean  for  all  27  classes  was  54.9  percent. 

To  visit  all  classes,  score  the  results,  and  arrange 
students  into  matched  pairs  according  to  reading  ability 
required  approximately  ten  days  to  complete. 

In  preparation  for  the  second  and  final  testing 
session,  the  two  members  of  each  matched  pair  were  randomly 
designated  pair  member  X or  pair  member  Y.  For  member 

X,  one  of  the  three  forms  of  the  cloze  test  was  prepared, 
the  three  forms  being  assigned  in  such  a manner  that  they 
were  equally  distributed  throughout  the  sample.  For  member 

Y,  a copy  of  the  completion-type  objective  test  was 
prepared. 

During  the  testing  session,  students  in  each  class 
were  told  initially  that  they  would  be  taking  one  of  two 
types  of  tests  that  were  being  administered,  either  a test 
similar  to  the  one  they  had  taken  in  testing  session  one 
(a  cloze  test),  or  a short- answer  completion  test  to  be 
taken  after  carefully  reading  a German  short  story.  The 
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tests  were  then  disseminated  and  students  were  admonished 
to  read  all  instructions  carefully  (see  Tests  1 1 A and  1 1 B 
in  Appendix  B).  To  administer  the  tests  required  approxi- 
mately 40  minutes  of  class  time. 

Scoring  Criteria 
The  Importance  of  Scoring 

A key  factor  effecting  the  establishment  of  an  objec- 
tive and  general i zabl e cloze  criterion  score  based  on 
information  gain  is  the  manner  in  which  the  data  are  scored. 
By  varying  the  scoring  procedures,  of  course,  the  cloze 
criterion  score  is  also  varied.  Therefore,  it  is  impera- 
tive to  select  the  most  reliable  scoring  method  possible 
and  then  ensure  that  it  is  consistently  administered. 

In  establishing  scoring  criteria,  several  factors  merit 
consideration:  On  the  one  hand,  because  German  teachers 
using  the  cloze  procedure  to  determine  the  appropriateness 
of  reading  selections  will  have  to  apply  the  same  scoring 
procedures  used  in  this  research  in  order  to  achieve  valid 
findings,  it  is  extremely  desirable  to  employ  a very  simple 
set  of  rules.  On  the  other  hand,  however,  if  reliability 
is  sacrificed  for  the  sake  of  scoring  ease,  the  primary 
purpose  for  having  established  an  objective  cloze  criterion 
score  will  have  been  eliminated.  How,  then,  does  one 
establish  the  most  appropriate  method  of  scoring? 

Some  Background  Information 

In  reviewing  tiic  literature,  it  is  immediately  ap- 
parent that  most  cloze  researchers  of  English  have 
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utilized  the  exact  scoring  method,  a procedure  that  gives 
credit  only  when  the  reader  fills  the  blank  with  precisely 
that  word  that  was  originally  deleted.  Such  decisions 
have  been  based  on  the  findings  of  investigators  such  as 
Taylor  (1953),  Rankin  (1957  and  1959),  Ruddell  (1963), 
and  Bormuth  (1965),  all  of  whom  compared  the  cloze  scores 
achieved  by  using  the  exact  scoring  method  with  those 
established  by  ■ counting,  both  synonyms  and  exact  matches 
and,  according  to  Rankin  (1965),  found  that  "for  measure- 
ment purposes,  the  more  tedious  and  subjective  procedure 
of  giving  credit  for  synonyms  is  not  worthwhile"  (p.  147). 
Despite  such  findings . however,  subsequent  researchers 
such  as  Darnell  (1968),  Bowen  (1969),  and  Oiler  and  Inal 
(1971),  who  have  experimented  with  nonnatives  taking  English 
cloze  tests,  have  preferred  to  accept  scoring  procedures 
in  which  at  least  partial  credit  was  given  for  contextu- 
ally acceptable  responses.  To  lend  support  to  such  actions 
when  administering  the  cloze  test  to  nonnative  speakers 
of  English,  Oiler  (1972b)  gave  three  cloze  tests  of 
varying  difficulty  to  398  foreign  students  that  had  also 
taken  the  UCLA  ESL  Placement  Examination  (2A  Revised). 

After  evaluating  each  of  the  cloze  tests  with  five  dif- 
ferent scoring  methods  and  then  correlating  the  findings 
with  the  ESL  Placement  Examination,  Oiler  (1973)  stated 
that  "the  data  supported  the  conclusion  that  with  non- 
native speakers  the  method  of  allowing  any  contextually 
acceptable  response  is  significantly  superior  to  t lie  exact 
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word  scoring  technique"  (p.  109). 

With  nonnative  speakers,  then,  there  exists  doubt 
as  to  the  superiority  of  the  exact  method  for  scoring 
English  cloze  tests.  Further  complicating  the  issue  in 
this  research,  however,  is  the  fact  that  German,  not  English, 
is  the  language  under  investigation.  Unlike  English,  which 
is  only  lightly  inflected,  German  is  a highly  inflected 
language.  Therefore,  in  scoring  a cloze  test,  how  is  one  to 
score  a blank  that  is  filled  with  the  correct  but  improp- 
erly inflected  word,  a problem  that  seldom  occurs  for 
either  native  or  nonnative  speakers  of  English?  In  order 
to  resolve  such  problems  and  determine  the  most  reliable 
scoring  criteria  for  this  research,  it  was  necessary  to 
conduct  a pilot  study. 

The  Pilot  Study 

The  specific  purpose  for  conducting  a pilot  study 
was  to  determine  the  answers  to  two  general  questions: 

1)  Which  scoring  method  is  more  reliable,  the  exact  or 
accepting  any  response  that  is  contextually  acceptable 
and  demonstrates  comprehension?  2)  In  dealing  with  a 
highly  inflected  language  such  as  German,  should  the 
ability  to  inflect  correctly  also  be  a scoring  consid- 
eration? Before  scoring  all  the  58-blank  cloze  matching 
tests  that  had  been  administered  in  testing  session  one, 
the  results  achieved  by  two  of  the  27  classes  first  were 
experimented  with  in  order  to  answer  the  above  two  ques- 
tions and  thereby  establish  the  most  appropriate  scoring 
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criteria. 

In  answering  question  one  and  determining  whether 
the  exact  scoring  method  was  better  than  the  acceptable- 
word  method,  cloze  tests  taken  by  a class  consisting  of 
the  most  advanced  German  students  were  chosen  for  eval- 
uation. The  selection  of  this  class,  which  contained 
16  students,  was  made  because  it  was  anticipated  that 
advanced  students  with  large  vocabularies  would  be  in- 
clined to  fill  the  cloze  blanks  with  a variety  of  acceptable 
words  and  at  the  same  time  make  only  minimal  inflectional 
errors.  Therefore,  the  cloze  results  achieved  by  this 
class  were  scored  twice  using  two  separate  scoring  pro- 
cedures. In  order  to  focus  on  the  effects  of  the  exact 
scoring  method  as  compared  with  the  acceptable-word  method, 
inflectional  considerations  were  treated  as  a constant, 
and  during  both  scorings  only  half  credit  was  awarded  when 
inflectional  mistakes  were  made.  During  the  first  scoring, 
only  the  exact  word  that  had  been  deleted  was  accepted 
as  correct.  If  the  exact  word  was  selected  but  improperly 
inflected,  however,  only  one-half  credit  was  awarded. 

During  the  second  scoring,  any  contextually  correct  word 
was  accepted.  Again,  however,  if  an  incorrect  inflection 
was  used,  only  one-half  credit  was  awarded.  The  relia- 
bilities of  these  two  scoring  methods  were  then  determined 
by  first  correlating  odd-  and  cvcn-numbcrcd  test  items 
and  then  correcting  the  resultant  correlations  by  applying 
the  Spearman- Brown  prophecy  formula.  By  using  the  exact 
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scoring  method,  a reliability  of  .66  was  achieved. 

Accepting  any  contextually  correct  word  boosted  reliability 
to  .88,  however,  and  suggested  that  this  was  a better 
scoring  procedure  for  this  research  project. 

In  answering  question  two  and  determining  whether 


at  least  some  inflectional  considerations  should  be  in- 
cluded as  part  of  the  scoring  criteria,  a class  of  15 
first-year  German  students  was  selected.  In  this  instance 
a beginning  class  was  chosen  because  it  was  anticipated 
that  unlike  the  advanced  class,  most  students  would  have 
very  limited  vocabularies  and  make  considerable  inflectional 
errors.  Again,  the  cloze  results  were  scored  twice  using 
two  separate  scoring  procedures.  Because  of  extreme 
vocabulary  limitations,  it  was  anticipated  that  most  students 
would  know  no  more  than  one  possible  word  for  each  blank. 

This  time,  therefore,  the  exact  scoring  method  was  treated 
as  the  constant  during  both  scorings,  and  only  the  exact 
word  was  accepted  as  correct.  To  get  at  inflectional  con- 
siderations, during  the  first  scoring  everything  had  to 


be  exact,  including  all  inflections,  in  order  to  receive 
credit.  During  the  second  scoring,  however,  one-half 
credit  was  awarded  if  an  improperly  inflected  exact  word 
was  supplied.  By  determining  the  correlations  in  the 
same  manner  as  had  been  done  with  the  class  of  advanced 
students,  it  was  found  that  requiring  correct  inflections 
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exact  word  incorrectly  inflected,  a reliability  of  .91 
was  achieved.  Therefore,  in  this  project  the  decision 
was  made  to  award  partial  (one-half)  credit  in  certain 
predetermined  cases  despite  incorrect  inflections. 

Rules  for  Scoring  the  Cloze  Test 

Based  on  the  findings  presented  above,  a set  of  ten 
rules  was  established  for  scoring  all  cloze  tests  used  in 
conjunction  with  this  research.  Although  even  higher 
reliabilities  undoubtedly  could  have  been  achieved  had 
even  more  detailed  scoring  procedures  been  used,  it  was 
felt  essential  to  keep  the  rules  relatively  general  and 
uncomplicated  in  order  not  to  overwhelm  and  thereby  dis- 
courage German  teachers  from  using  the  findings  of  this 
research  as  a tool  for  assessing  reading  difficulty. 

Rules  established  for  scoring  all  cloze  tests  used  in 
conjunction  with  this  research  include: 

1.  Assign  two  points  to  every  cloze  blank. 

2.  Accept  as  semantically  correct  the  exact  word 
deleted,  a synonym  of  the  exact  word,  or  any  con- 
textually acceptable  substitute  that  demonstrates 
comprehension. 

3.  Subtract  one  point  for  an  incorrect  noun-modifier 
or  noun  ending. 

4.  Subtract  one  point  for  an  incorrect  tense  or  tense 
form. 

5.  Subtract  one  point  for  an  incorrect  plural  form. 

6.  Subtract  one  point  for  an  article  (definite  or 
indefinite)  that  does  not  agree  in  gender,  number, 
and  case  with  the  article  that  was  deleted. 

7.  Unless  contextually  acceptable,  award  no  credit 
if  either  an  indefinite  article  is  used  to  replace 
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a definite  article  or  a definite  article  is  used  to 
replace  an  indefinite  article. 

8.  Where  a preposition  is  called  for,  accept  as  cor- 
rect only  the  preposition  that  was  deleted  except  in 
cases  where  two  or  more  prepositions  can  be  considered 
semantically  correct.  For  Example,  "hr  sitzt  ini  or 
hinter  dem  Tisch." 

9.  In  all  inflectional  instances  not  described  above, 
accept  only  an  exact  replacement. 

10.  Convert  each  raw  cloze  score  into  a percentage 
by  dividing  the  total  number  of  correct  points 

by  the  total  number  of  available  points. 

Rules  for  Scoring  the  Completion  Test 

Because  the  objective  of  establishing  a cloze  criterion 
score  for  reading  based  on  information  gain  involves  a 
comparison  between  cloze  scores  and  completion-type  ob- 
jective test  scores  obtained  from  the  same  reading  passage 
by  matched  students  of  German,  it  is  essential  that  the 
rules  for  scoring  the  completion  test  also  be  as  objective 
as  possible.  In  developing  rules  for  scoring  the  comple- 
tion test  in  this  research,  guidelines  laid  down  by  Bormuth 
(1971)  in  his  Study  Two  were  heavily  drawn  from.  Sim- 
plistically  stated,  the  scoring  procedure  involved  classi- 
fying responses  as  being  either  derivable  or  not  derivable 
from  the  reading  passage.  Those  responses  that  were  de- 
rivable were  scored  correct;  those  that  were  not,  were 
marked  incorrect. 

In  determining  which  responses  were  derivable  from 
the  reading  selection  and  thereby  correct,  all  answers 
were  examined  for  their  relationship  to  the  generic  response, 
which  was  defined  as  cither  the  word(s),  phrase,  or  clause 
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replaced  by  the  wh-  words  to  form  the  questions.  In  answering 
the  questions,  it  was  possible  for  a correct  answer  to 
have  either  a formal  or  a semantic  relationship  to  the 
generic  response.  If  an  answer  was  formally  identical 
to  the  generic  response,  naturally  it  was  scored  correct. 

For  example,  if  the  generic  response  that  had  been  deleted 
was  "the  man,"  the  answer  "the  man"  would  be  considered 
correct.  If  the  answer  was  formally  identical  to  only 
part  of  the  generic  response,  only  the  word  that  was  the 
grammatical  head  of  the  generic  response  was  required  in 
order  for  the  answer  to  be  scored  correct.  For  example, 
with  the  generic  response  "the  tall  man,"  either  "the  tall 
man"  or  "the  man"  would  be  correct.  If  the  generic  response 
was  a coordinated  structure,  only  one  of  the  coordinated 
structures  was  required.  For  example,  with  the  generic 
response  "golf  and  tennis,"  either  "tennis  and  golf," 

"golf,"  or  "tennis"  would  be  correct. 

In  addition  to  formal  considerations,  semantic 
relationships  between  the  generic  response  and  the  answer 
were  also  considered.  Any  answer  with  a correct  semantic 
relationship  to  the  generic  response  was  counted  correct. 

For  example,  if  the  correct  generic  response  was  "in- 
struments," "tools"  was  also  accepted  as  correct. 

Finally,  because  the  objective  of  the  completion  test 
was  to  determine  how  much  the  reader  actually  understood 
in  order  to  make  a valid  point  of  comparison  for  the  cloze 
test,  those  students  taking  the  objective  test  were  allowed 
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to  answer  in  either  German  or  English.  Justification  for 
this  decision  was  based  on  the  realization  that  if  a reader 
knew  the  correct  answer  but  could  not  think  of  the  German 
word  and  therefore  left  the  answer  blank,  maximum  compre- 
hension would  not  have  been  reflected  and  the  ultimate 
validity  of  the  findings  would  have  been  reduced.  There- 
fore, English  answers  were  accepted  on  the  completion  test 
although  German  responses  were  encouraged. 

Statistical  Analysis 

In  order  to  establish  an  objective  and  rational  cloze 
criterion  score  based  on  information  gain,  it  is  necessary 
to  determine  some  meaningful  relationships  by  comparing 
the  cloze  test  with  an  instrument  that  effectively  measures 
information  gain.  When  conventional  completion  tests  are 
objectively  constructed  according  to  a set  of  predetermined, 
replicable  rules,  such  tests  can  be  used  as  the  measure  of 
information  gain.  To  determine  the  exact  relationship 
between  the  two  test  types,  however,  it  is  necessary  to 
subject  the  data  to  a rigorous  statistical  analysis. 
Therefore,  after  having  administered  and  scored  both  the 
cloze  test  and  the  completion  test  according  to  standards 
already  outlined,  all  results  obtained  from  the  105  matched 
pairs  of  readers  were  analyzed  by  means  of  regression 
analysis  in  order  to  predict  information  gain  from  the 
cloze  scores  and  thereby  establish  an  objective  cloze 
criterion  score.  In  the  analysis,  cloze  scores  were  treated 
as  the  independent  variable,  while  t he  objective  test 
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scores  were  treated  as  the  dependent  variable.  All  analyses 
were  performed  either  on  an  IBM  370  computer  located  at 
The  Ohio  State  University  (OSU)  or  on  a Burroughs  6700 
computer  located  at  the  United  States  Air  Force  Academy 
(USAFA) . All  analyses  were  accomplished  using  either  the 
statistical  program  designated  "Statistical  Analysis 
System"  (SAS)  by  Barr  et  al  (1976)  or  the  program  designated 
"Statistical  Package  for  the  Social  Sc iences" (SPSS)  by 
Nie  et  al  (1975).  All  plotting  presented  in  this  research 
was  done  using  the  Calcomp  Plotting  System  at  USAFA. 
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RESULTS  AND  DISCUSSION 

The  intent  of  Chapter  IV  is  to  analyze  and  discuss 
the  findings  of  this  research.  To  assure  clarity  and 
continuity,  the  chapter  is  divided  into  five  sections: 

1)  a brief  discussion  of  regression  analysis,  the  primary 
statistical  tool  used  to  analyze  the  findings;  2)  a look 
at  the  reliabilities  of  the  two  test  types;  3)  an  exam- 
ination of  why  and  how  the  data  were  modified  in  pre- 
paration for  analysis;  4)  the  actual  analysis  of  the  data 

with  a discussion  of  the  findings;  and  5)  answers  to  the 

/ 

three  research  questions. 

Regression  Analysis 

When  two  separate  measurements,  say  X and  Y,  have 
been  made  on  a group  of  individuals,  it  is  frequently 
desirable  to  study  these  measurements  simultaneously  in 
order  to  determine  their  relationship.  To  do  so,  several 
different  procedures  can  be  employed.  For  example,  one 
can  construct  a simple  scnttcrgrnm,  which  employs  two 
perpendicular  axes,  and  plot  each  individual's  scores. 

By  examining  t he  pattern  of  the  plots,  one  can  sometimes 
determine  if  and  how  the  two  measurements,  generally 
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referred  to  as  variables,  are  related.  If  more  sophisti- 
cation and  precision  are  desired,  the  degree  of  relation- 
ship between  the  two  measurements  can  be  determined  mathe- 
matically by  computing  a statistic  referred  to  as  the 
correlation  coefficient  (r).  Should  one  he  interested  in 
using  one  of  the  measurements  (the  independent  X variable) 
to  explain  or  predict  the  other  measurement  (the  dependent 
Y variable),  a statistical  procedure  referred  to  as  re- 
gression analysis  is  employed.  The  notion  of  regression 
analysis  is  closely  related  to  that  of  correlation.  In 
fact,  the  "r"  used  to  indicate  the  correlation  coefficient 
means  "regression."  Depending  on  the  number  of  independent 
variables  involved,  regression  analysis  is  traditionally 
divided  into  two  broad  categories,  simple  and  multiple 
regression.  In  both  instances  the  relationship  between 
the  independent  variable(s)  and  the  dependent  variable 
can  be  either  linear  or  nonlinear.  Because  the  objective 
of  this  research  is  to  examine  only  one  independent 
variable,  the  cloze  test,  the  topic  of  multiple  regres- 
sion will  not  be  dealt  with. 

Simple  Linear  Regression 

In  simple  regression  a single  independent  variable 
X is  used  to  explain  or  predict  some  dependent  variable 
Y.  To  make  this  prediction,  one  must  compute  the  best 
fitting  regression  curve  of  Y on  X,  i.e.,  that  curve  or 
line  that  accounts  for  the  maximum  amount  of  dependent 
variable  variance.  Of  the  various  types  of  curves  that 
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can  be  fitted  to  the  data,  the  one  most  frequently  used 
is  the  linear  or  straight  line,  probably  because  it  is 
the  simplest  to  compute.  The  basic  equation  used  to 
express  and  compute  simple  linear  regression  is: 

Y«  = a + bX 

In  this  equation  X represents  the  various  values  of  the 
independent  variable,  Y'  is  the  predicted  dependent  vari- 
able, "a"  is  the  intercept  constant  (that  point  at  which 
the  regression  line  intercepts  the  Y axis)  , and  "b"  is 
the  regression  coefficient  (that  number  which  identifies 
the  slope  of  the  regression  line). 

Curvilinear  Regression 

If  for  some  reason  a researcher  is  led  to  believe 
that  the  relationship  between  X and  Y is  something  other 
than  linear,  possibly  because  of  some  underlying  theory 
or  simply  by  examining  the  data  plots  on  a scattergram, 
it  then  becomes  expedient  to  try  and  fit  the  data  to  some 
type  of  nonlinear  mathematical  model,  such  as  a trigono- 
metric or  logarithmic  function.  For  example,  if  a scatter- 
gram of  the  data  revealed  plots  somewhat  resembling  those 
found  in  Figure  there  would  be  a strong  indication  that 
the  relationship  between  the  variables  resembled  a logarithmic 
function.  Logically,  therefore,  one  would  abandon  fitting 
the  data  to  a straight  line  and  employ  the  following  equation: 

Y'  ® a + b LOGX 

which  first  causes  the  independent  variable  values  to  be 
transformed  into  their  logarithmic  function  before  being 
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Figure  3.  Simulated  scattergram  of  nonlinear  data  points. 


multiplied  by  the  regression  coefficient  "b . " 

In  addition  to  fitting  data  by  using  some  type  of 
trigonometric  or  logarithmic  function,  a different  type 
of  nonlinear  regression  referred  to  as  polynomial  regression 
analysis  is  often  employed  if  some  type  of  nonlinear  trend 
in  the  data  points  is  suspected.  In  performing  polynomial 
regression,  the  independent  variable  is  raised  to  some 
power,  with  the  highest  power  representing  the  degree  of 
the  polynomial  equation.  As  an  example,  the  equation 

Y'  = a + bjX  ♦ b2X2 

is  a second- degree  polynomial,  often  referred  to  as  a 
quadratic  equation,  because  the  independent  variable  X has 

i 

been  raised  to  the  second  power.  The  order  of  the  poly- 
nomial equation  indicates  the  maximum  number  of  bends  that 
can  occur  in  the  regression  curve.  For  example,  because 
the  above  equation  is  a second-degree  polynomial  containing 
two  terms  (X  and  X ),  it  describes  a single  bend  in  the 
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regression  curve  and  would  provide  an  excellent  fit  for 

¥ 

data  similar  to  that  found  in  Figure^^  Similarly,  a 
third-degree  polynomial  equation  containing  three  terms 
(X,  X2,  and  X3)  provides  for  two  bends  in  the  regression 
curve,  and  so  on.  According  to  Kerlinger  and  Pedhazur 
(1973),  "the  highest  order  that  any  given  (polynomial) 
equation  may  take  is  equal  to  K — 1,  where  K is  the  number 
of  distinct  values  in  the  independent  variable"  (p.  209). 
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Figure  4.  Simulated  scattergram  of  nonlinear  data  points. 


Although  the  highest  order  polynomial  equation  would 
provide  for  K — 1 bends  in  the  regression  curve  it  generated, 
it  would  nevertheless  have  little  predictive  value  and  be 
of  little  use  to  the  researcher.  In  this  regard,  Dunn  and 
Clark  (1974)  indicate: 

It  is  possible  to  approximate  any  regression 
curve  us  closely  as  is  desired  by  using  a polynomial 
of  sufficient  high  degree.  However,  in  fitting  a 
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polynomial  to  a set  of  data,  we  are  not  just  inter- 
ested in  obtaining  a curve  that  lies  very  close  to 
the  sample  data  points.  Rather,  we  are  attempting 
to  obtain  a curve  that  approximates  the  population 
regression  curve  (line  of  population  means).  If  our 
primary  interest  were  in  fitting  the  curve  to  the 
data  points,  we  could  actually  fit  a polynomial  of 
sucli  high  degree  that  it  would  pass  exactly  through 
every  data  point.  A straight  line  (polynomial  of 
degree  1)  can  be  passed  througli  any  two  points;  a 
quadratic  (degree  2)  can  be  passed  through  any  three 
points;  a cubic  (degree  3)  can  be  passed  through  any 
four  points;  an  so  on.  Usually  K,  the  degree  of  the 
fitted  polynomial,  is  chosen  to  be  2 or  3;  certainly 
K should  be  quite  small  compared  with  n,  the  sample 
size  (p.  295). 


Test  Reliabilities 


Prior  to  submitting  the  data  to  the  computer  for  re- 
gression analysis,  the  reliabilities  foi  both  the  cloze 
test  and  the  completion  test  were  computed  to  ensure  that 
both  test  types  were  stable,  consistent,  and  accurate. 


The  reliability  of  each  of  the  two  test  types  was  determined 
by  first  correlating  all  of  the  odd-numbered  items  in  each 
test  with  all  of  the  even-numbered  items  and  then  correcting 
(boosting)  the  resultant  correlations  by  using  the  Spear- 
man— Brown  prophecy  formula. 

Although  great  pains  had  been  taken  in  developing 
the  completion- type  test,  as  was  expected,  because  of 
the  cloze  test’s  simplicity  of  construction,  a higher 
reliability  was  realized  on  the  cloze  test.  For  the  com- 
pletion test  a reliability  of  .84  was  achieved,  while  on 
the  cloze  test  a reliability  of  .95  was  obtained.  Because 
on  the  cloze  exam  three  different  test  forms  had  been 
constructed  and  administered  in  order  to  counterbalance 
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any  effects  that  might  arise  should  the  items  in  one  form 
differ  in  difficulty  from  the  other  two  forms,  both  the 
mean  and  reliability  were  computed  for  each  of  the  three 
cloze  forms.  As  shown  in  Table  6,  there  were  indeed 
measurable  differences  in  the  three  different  forms  of 
the  cloze  test- -di f ferences  that  could  have  affected  the 
findings  had  only  one  form  been  used  instead  of  three. 


TABLE  6 

CLOZE-TEST  CORRELATIONS  AND  MEANS 


Form 

Correlation 

Mean  (1) 

1 

.96 

50.6 

2 

.93 

58.5 

3 

.96 

50.8 

All  3 

.95 

53.2 

Data  Modification 

In  preparing  the  data  for  analysis,  all  cloze  scores 
were  first  adjusted  in  order  to  correct  them  for  a statis- 
tical bias  that  had  been  pointed  out  by  Bormuth.  In  dis- 
cussing the  regression  models  he  had  used  in  his  Study 
One,  Bormuth  (1971)  says  the  following: 
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The  regression  models  used  in  these  studies  do 
not  take  into  account  the  error  of  measurement  in  the 
independent  variable,  the  cloze  scores.  This  error 
of  measurement  causes  a student's  observed  cloze  score 
to  lie  farther  from  the  mean  than  its  true  value, 
the  mean  of  the  values  that  would  be  observed  if  he 
could  be  administered  the  test  an  infinite  number 
of  times.  The  effect  of  this  bias  would  generally 
be  to  associate  a gain  score  with  an  observed  score 
that  lay  farther  from  the  cloze  mean  than  the  true 
cloze  score  with  which  it  is  associated,  thus  stretch- 
ing the  curves  at  the  extremes.  Since  this  bias  would 
influence  the  level  at  which  a performance  criterion 
was  placed,  it  was  removed  by  correcting  the  cloze 
scores  to  their  true  scores  (p.  37). 

In  order  to  remove  the  above-mentioned  bias,  all  cloze 
scores  were  transformed  into  what  Bormuth  refers  to  as 
their  "true"  scores  by  using  the  formula: 

Xt  = rX  + (1  - r) X 

where  Xt  represents  the  student's  true  score,  "r"  the 
reliability  of  the  test,  X the  student's  observed  score, 
and  )(  the  test  mean. 

To  obtain  the  transformed  score  X^  for  analysis, 
each  of  the  105  observed  cloze  test  scores  X were  entered 
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a scattergram  of  the  data  plots  was  obtained  in  order  to 
make  a preliminary  visual  analysis.  As  Figure  ^^s hows , 
at  least  the  middle  portion  of  the  data  plots  appeared  to 
be  generally  linear.  As  a beginning,  therefore,  it  was 
decided  to  fit  the  data  with  a simple  linear  regression 
model . 

Using  the  regression  procedure  contained  in  SAS,  the 
following  linear  regression  equation  was  derived: 

Y'  = 36.05553314  + .56959232X 
which  yielded  the  regression  line  found  in  Figure  -Sr 
That  the  linear  regression  model  fit  the  data  reasonably 
well  and  that  there  was  a statistically  significant  re- 
lationship between  the  two  test  types  was  substantiated 
both  by  an  analysis  of  variance  (ANOVA) , the  findings  of 
which  are  contained  within  Table  7,  and  by  the  fact  that 
the  correlation  between  the  two  variables  was  computed 
to  be  .73.  This  statistic  squared  (r  * .53139)  yielded 
the  coefficient  of  determination,  which  indicated  that 
just  over  53  percent  of  the  variance  detected  in  the 
completion- test  scores  could  be  accounted  for  by  the 
cloze  scores. 

Polynomial  Regression  Analysis 

Despite  the  fact  that  a simple  linear  regression  model 
fit  the  data  points  rather  well  and  accounted  for  a con- 
siderable amount  of  the  variance,  a plot  of  the  regres- 
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sion  line  through  the  data  points  (see  Figure  'M)  strongly 
suggested  that  using  a more  complex,  curvilinear  model 


Figure  5.  Scat-tergram  of  actual  data  points 


TABLE 


122 

would  most  likely  account  for  additional  variance  and 
provide  for  a better  fit. 

Suspecting  that  the  line  of  best  fit  would  likely 
contain  several  bends,  the  decision  was  made  to  fit  the 
data  using  a polynomial  regression  model.  In  choosing  a 
statistical  package  for  polynomial  regression  analysis, 

SAS  was  again  chosen,  primarily  for  two  reasons:  1)  The 
SAS  package  provides  for  two  different  types  of  F-tests 
of  statistical  significance:  a)  a sequential  F-test 
(referred  to  by  SAS  as  the  Type  I Sum  of  Squares),  which 
adds  the  various  powers  (terms  of  the  equation)  of  the 
independent  variable  sequentially  one  by  one  into  the 
polynomial  regression  equation,  and  b)  a partial  F-test 
(referred  to  by  SAS  as  the  Type  IV  Sum  of  Squares),  which 
treats  each  of  the  powers  of  the  independent  variable  as 
though  it  were  added  to  the  equation  last.  According 
to  Draper  and  Smi';h  (1966),  the  partial  F-test  is  im- 
portant because  it  allows  the  researcher  "to  sec  the  rela- 
tive effects  of  each  variable  in  excess  of  the  others" 

(p.  72). 

2)  The  SAS  package,  rather  than  simply  supplying  the 
F statistic,  the  significance  of  which  generally  must  be 
determined  by  comparing  it  with  a tabled  value,  also 
provides  the  computed  confidence  level  corrected  to  the 
fourth  decimal  place. 

In  using  the  polynomial  regression  model,  the  inde- 
pendent variable,  cloze-test  scores  were  successively 
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raised  in  power  (b^X  + b2X2,  then  b^X  + b£X^  + b^X^,  etc.) 
from  the  second-  up  to  the  sixth-degree  polynomial.  At 
each  of  the  separate  powers,  a regression  analysis  was 
run  in  order  to  determine  which  of  the  different  polynomial 
regression  equations  would  best  fit  the  data.  In  analyzing 
the  resultant  sequential  F-tests  computed  at  each  of  the 
different  powers  (X2  through  X6),  totally  insignificant 
F-values  were  obtained  in  every  instance  except  when  a 
fifth-degree  polynomial  was  derived.  At  that  level  the 
computed  F-value  proved  to  be  much  more  meaningful,  resulting 
in  a conficence  level  of  just  over  .08  (see  Table  8).  As 
a confirmation  of  the  significance  of  the  fifth-degree 
polynomial,  findings  from  the  partial  F-test  (see  Table  9) 
did,  in  fact,  indicate  that  each  of  the  five  separate  b- 
coefficient  terms  of  the  independent  variable  in  the  equation 
had  a significant  effect  when  added  to  the  equation  last. 

As  Table  10  indicates,  an  analysis  of  the  variance  of  the 
overall  regression  of  the  fifth-degree  polynomial  reflected 
an  extremely  significant  F-value.  Table  11,  a summary 
of  all  the  correlational  relationships,  reflects  a high 
multiple  correlation  of  more  than  .74  between  all  the 
components  of  the  fifth-degree  polynomial  and  the  depend- 
ent variable,  the  completion-type  comprehension  test,  and 
further  indicates  that  such  a regression  model  accounts 
for  over  55  percent  of  the  variance  detected  in  the  depend- 
ent variable. 
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As  a result  of  the  statistical  significance  of  the 
fifth-degree  polynomial,  the  decision  was  made  to  use  the 


following  equation  to  fit  the  data 


Substituting  the  coefficients  derived  from  the  fifth-degree 
polynomial  regression  analysis  yielded  the  following 


equation 


520828X 


000 10  399Xh  + . 000000368X 


+ . 0109 1 11X 


which  when^plotted  resulted  in  the  regression  curve  found 
in  Figure  In  addition  to  accounting  for  more  of  the 

variance,  Figure  ,-I^shows  that  the  fifth-degree  polynomial 
curve,  when  compared  with  the  line  resulting  from  the 
linear  equation  (see  Figure  Zr.  appears  to  fit  the  data 
considerably  better. 

Stepwise  Regression  Analysis 


Although  the  above  findings  indicated  that  the  data 
points  could  be  fitted  well  with  a fifth-degree  polynomial 
regression  model,  it  was  nevertheless  decided  to  construct 
a second,  different  curvilinear  model  for  purposes  of 
comparison  and  confirmation.  Instead  of  forcing  all  of  the 
b-coef f icient  terms  of  the  polynomial  equation  into  the 
regression,  however,  as  had  been  done  in  constructing  the 
above  fifth-degree  polynomial  model,  it  was  decided  to 
use  a stepwise  regression  procedure.  In  stepwise  regres- 
sion the  independent  variable  and  its  various  nonlinear 
transformations  that  are  selected  for  examination  are 
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Figure  9.  Fifth-degree  polynomial  regression  curve 
ted  through  the  specific  data  points. 
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only  entered  into  the  equation  if  they  meet  certain 
statistical  criteria.  In  addition,  the  order  of  their 
inclusion  is  determined  not  by  the  researcher,  but  by  the 
respective  contribution  that  each  one  makes  in  explaining 
variance.  According  to  Nie  et  al  (1975): 

The  variable  that  explains  the  greatest  amount 
of  variance  in  the  dependent  variable  will  enter 
first;  the  variable  that  explains  the  greatest 
amount  of  variance  in  conjunction  with  the  first  will 
enter  second,  and  so  on.  In  other  words,  the  variable 
that  explains  the  greatest  amount  of  variance  un- 
explained by  the  variables  already  in  the  equation 
enters  the  equation  at  each  step.  And  one  or  more 
of  the  variables  may  never  be  entered  into  the  re- 
gression equation  if  the  statistical  criteria  are 
not  met  (p . 345) . 

In  deciding  which  nonlinear  transformations  of  the 

independent- variable  data  to  use  in  carrying  out  stepwise 

regression  analysis,  it  seemed  appropriate  that  the  various 

2 3 

power  functions  of  the  independent  variable  (X  , X , etc.) 

should  again  be  employed,  at  least  up  to  the  fifth  power 

where  high  confidence  levels  had  been  derived  with  the 

first  curvilinear  regression  model.  In  addition,  however, 

it  was  decided  to  use  two  other  types  of  nonlinear  trans- 

/& 

formations,  the  logarithmic  (LOG)  function  (see  Figure^) 

/' 

and  the  arc  tangent  (ARC)  function  (see  Figure  4-00,  both 

of  which  appeared  might  provide  a good  fit  for  the  data 

and  thereby  account  for  additional  variance. 

Therefore,  using  the  regression  procedure  contained 

in  SPSS,  the  independent- variable  scores  (X)  and  six  non- 

2 3 3 5 

linear  transformations  of  those  scores  (X  , X , X , X , 
LOGX,  and  ARCX)  were  submitted  to  a stepwise  regression 
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analysis.  As  was  expected,  the  first  value  to  be  selected 
was  the  nontransformed  value  X because,  as  had  been  dis- 
covered when  employing  the  simple  linear  model,  it  alone 
could  account  for  just  over  53  percent  of  the  variance 
in  the  dependent  variable,  complet ion-test  scores.  In 
order  of  their  selection  into  the  regression,  those  trans- 
formations of  X that  were  able  to  account  for  statistically 
significant  additional  amounts  of  variance  included: 

X5,  X4,  and  ARCX. 

As  Table  12  indicates,  an  analysis  of  the  variance 
of  the  overall  regression  containing  the  above  four  vari- 
ables again  reflects  an  extremely  significant  F-value. 

Table  13  summarizes  the  correlational  relationships. 

Although  the  total  amount  of  variance  (r-square)  that 
this  model  accounts  for  is  also  very  high,  a comparison 
with  findings  resulting  from  the  fifth-degree  polynomial 
model  (see  Table  11,  p.  127)  indicates  that  that  model  is 
slightly  better,  accounting  for  over  one  percent  more  of 
the  variance  detected  in  the  dependent  variable. 

Despite  the  fact  that  the  stepwise  regression  model 
was  slightly  less  efficient  than  the  fifth-degree  polynomial 
model,  the  findings  were  nevertheless  plotted  as  a means 
of  comparison.  Using  the  coefficients  derived  from  the 
stepwise  regression  yielded  the  following  equation: 

Y'  - .08866  - . 37775X  - .00033X5  + .0036SX4  + 5.01913ARCX 


which  whon  plotted  resulted  in  the  curve  found  in  Figure 
As  Figure  shows , both  types  of  regression  models 


& 


Figure  12.  Plot  of  four-term  stepwise  regression  curve 
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generate  curves  that  are  quite  similar  except  at  the  ex- 
treme upper  end  where  the  arc  tangent  transformation  of  X 


apparently  forces  the  regress  ion  curve  of  the  stepwise 
model  down.  I-igurc  yff  indicates  that  both  curves  fit  the 
data  points  well,  although,  as  might  be  expected,  the 
fifth-degree  polynomial  cuofe  appears  to  fit  slightly 
better.  Finally,  Figure  ^T^depicts  the  regression  lines 
of  all  three  models  (simple  linear,  fifth-degree  poly- 
nomial, and  four-term  stepwise)  plotted  through  the  data 
points.  In  the  central  region  all  three  models  fit  well. 
Again,  however,  overall,  the  fifth-degree  polynomial  model, 
which  accounts  for  the  largest  amount  of  the  variance, 
appears  to  provide  the  best  fit.  It  is  therefore  this 
model  that  will  be  used  in  drawing  the  various  conclusions 
of  this  research. 


The  Research  Questions 


Having  analyzed  the  findings  and  identified  the  re- 
gression model  that  best  fits  the  data  to  a regression 
curve  for  analysis,  it  is  now  appropriate  to  address  the 
three  research  questions  initially  stated  in  Chapter  I. 
Question  One 

1.  Is  there  a relationship  between  the  cloze  test 
and  information  gain,  as  measured  by  a specially  constructed, 
short-answer  completion  test,  when  both  test  types  are 
prepared  from  a single  German  prose  reading  selection, 
selectively  administered  to  a carefully  chosen  group  of 
col lege- level  nonnativc  speakers  of  German,  and  then 
statistically  analyzed? 

Based  on  the  findings  of  this  research,  there  is  a 
significant,  measurable  relationship  between  the  cloze 
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through  the  specific  data  points. 
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test  and  information  gain,  as  measured  by  the  specially 
constructed,  short-answer  completion  test.  The  simple 
correlation  between  the  two  tests  is  .73,  with  over  53 
percent  of  the  coinplet ion- test  variance  being  accounted 
for  by  the  cloze  test.  Additionally,  by  transforming 
the  cloze-test  values  to  their  fifth  power  (X5),  a multiple 
correlation  of  .75  is  achieved.  Such  a transformation 
results  in  nearly  56  percent  of  the  completion-test  vari- 
ance being  accounted  for. 

Question  Two 

2.  If  there  is  a relationship  between  the  two  test 
types,  will  it  be  nonlinear,  as  Bormuth's  work  in  English 
suggests,  when  applied  to  nonnative  speakers  of  German? 

In  confirmation  of  Bormuth's  (1962  and  1971)  cloze 
research  in  English,  these  findings  also  indicate  that 
the  relationship  between  the  cloze  test  and  the  comple- 
tion-type objective  test  is  nonlinear  when  applied  to 
nonnative  speakers  of  German,  a statement  supported  by 
the  fact  that  a nonlinear  (fifth-degree  polynomial)  re- 
gression model  generated  the  regression  curve  that  best 
fit  the  data.  Of  equal  significance,  however,  is  the  fact 
that  t\^  plot  of  the  fifth-degree  polynomial  curve  (see 
Figure /7,  p.  129)  generally  confirms  Bormuth's  con- 
tention that  a cloze  test  discriminates  over  a broader 
range  of  reading  abilities  than  does  an  objective  test. 

Such  a phenomenon  is  especially  evident  in  the  lower 
ranges  (20  percent  to  40  percent)  of  the  cloze  test  where 
the  slope  of  the  regression  curve  is  very  close  to  zero. 
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The  fact  that  the  predicted  completion  test  scores  in 
this  cloze  range  stand  at  slightly  over  50  percent  is  not 
necessarily  indicative  that  large  amounts  of  significant 
information  have  been  gained  at  this  level  as  a result 
of  reading.  On  the  contrary,  such  a score  on  the  com- 
pletion test  used  in  this  study  probably  represents  only 
spotty  understanding  of  isolated  information  spread  through- 
out the  reading  selection.  This,  because  the  test  writing 
algorithm  used  to  construct  the  completion  test  generated 
38  questions,  an  extremely  large  amount  for  such  a short 
passage.  As  a result,  many  of  the  questions  are  very 
easy,  some  probably  as  easy  as  some  of  the  easiest  cloze 
blanks,  and  as  with  the  cloze  test,  can  be  answered  with- 
out real  understanding  of  the  overall  passage.  As  the 
regression  curve  shows,  the  cloze  test  does  better  dis- 
criminate, especially  in  the  lower  ranges,  a fact  strongly 
suggesting  that  an  objective  German  cloze  criterion  score 
can  be  established  that  docs  indeed  reflect  that  cloze 
percentage  score  where  significant  information  begins  to  be 
won  as  a result  of  reading. 

Question  Three 

3.  If  the  relationship  between  the  two  test  types 
is  nonlinear,  can  a rational,  objective  cloze  criterion 
score  that  is  valid  for  any  German  prose  reading  selection 
be  identified,  which  reflects  that  point  on  the  cloze 
test  at  which  nonnative  readers  of  German  begin  gaining 
substantial  information  as  a result  of  reading  German 
prose? 

Having  shown  that  the  cloze  test  and  the  completion- 
type  objective  test  arc  highly  related  in  a manner  best 
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described  by  a fifth-degree  polynomial  regression  curve, 
steps  can  now  be  taken  to  identify  an  objective  cloze 
criterion  score.  Because  of  Bormuth's  (1971)  preliminary 
cloze  research,  which  substantiated  the  psychometric  as- 
sumption of  regression  identity  that  regression  curves 
will  not  significantly  change  regardless  of  the  number 
of  reading  passages  used,  the  cloze  criterion  score  to  be 
established  in  this  research  can,  in  fact,  be  considered 
valid  for  any  German  prose  reading  selection. 

As  a first  step  in  establishing  the  cloze  criterion 
score,  a visual  analysis  of  the  regression  curve  (see 
Figur erf  indicated  that  the  score  might  be  placed  at  about 
35  percent,  a point  where  the  slope  of  the  regression 
curve  appears  to  begin  to  rise.  In  order  to  be  more  precise, 
however,  the  exact  mathematical  point  of  inflection  was 
computed,  i.e.,  that  point  along  the  lower  end  of  the 
regression  curve  where  the  curvature  reverses  concavity. 

To  make  this  computation,  the  fifth-degree  polynomial 
regression  equation  was  subjected  to  computer  analysis. 

By  taking  the  second  derivative  of  the  equation,  setting 
it  equal  to  zero,  and  solving  for  the  appropriate  cloze 
score,  it  was  mathematically  determined  that  the  regression 
curve  reverses  concavity  at  exactly  30.1  percent  on  the 
cloze  test.  Technically,  therefore,  this  cloze  score 
(30  percent)  can  be  considered  to  be  the  absolutely  lowest 
point  at  which  significant  information  may  begin  to  be 
gained  as  measured  by  the  cloze  test. 
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In  addition  to  establishing  the  mathematical  point 
of  inflection,  which  identifies  the  lowest  point  at  which 
a cloze  criterion  score  might  be  set,  it  was  also  felt  that 
an  upper  point  also  would  be  of  value  by  thus  allowing  the 
researcher  to  choose  within  the  range  of  the  two  scores. 
Because  the  slope  of  the  regression  curve  appears  to  ap- 
proach linearity  in  its  middle  ranges,  it  was  decided 
to  calculate  that  point  along  the  regression  curve  where 
its  slope  becomes  equal  to  the  slope  of  a straight  line 
plotted  through  the  data  points.  Using  the  b-coef f icient 
(.5659232)  that  had  been  derived  from  the  simple  linear 
regression  to  determine  that  model's  slope  through  the 
data,  the  fifth-degree  polynomial  equation  was  again  entered 
into  the  computer.  By  taking  the  first  derivative  of 
the  equation,  setting  it  equal  to  .57,  the  b- coefficient 
of  a straight  line,  and  again  solving  for  the  appropriate 
cloze  score,  it  was  mathematically  determined  that  the 
slope  of  the  polynomial  regression  curve  is  equal  to  the 
curve  of  a straight  line  at  42.8  percent  on  the  cloze 
test.  Technically  again,  therefore,  this  score  (43  per- 
cent) can  be  considered  to  be  the  absolute  maximum  at  which 
significant  information  may  begin  to  be  gained  as  measured 
by  the  cloze  test.  Reader's  scoring  at  or  above  this 
point  definitely  are  gaining  substantial  amounts  of  in- 
formation as  a result  of  their  reading. 

Having  established  a range  (30  percent  through  43 
percent)  on  the  cloze  test  within  which  significant 
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information  begins  to  be  gained,  the  final  step  was  simply 
to  establish  which  specific  score  within  that  range  should 
be  selected  as  the  cloze  criterion  score.  This  was  ac- 
complished by  considering  the  standard  error  of  estimate 
along  the  regression  curve.  Because  the  standard  error 
was  10.3,  it  was  decided  to  establish  the  cloze  criterion 
score  at  40  percent,  approximately  one  standard  error 
above  the  absolute  low  point  of  30  percent.  By  setting 
the  cloze  criterion  score  at  this  point,  one  could  be 
sure  of  being  correct  at  least  68  percent  of  the  time, 
especially  if  the  cloze  results  were  based  on  a class 
mean  rather  than  an  individual  score. 

As  a result  of  the  above  reasons,  40  percent  is  the 
cloze  score  that  has  been  chosen  to  identify  that  point 
at  which  significant  information  gain  begins  to  occur 
when  any  German  cloze  test  is  scored  in  compliance  with 
the  ten  rules  presented  in  Chapter  III  (see  p.  104). 
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CHAPTER  V 

SUMMARY,  LIMITATIONS, 

IMPLICATIONS,  AND  RECOMMENDATIONS 

Chapter  V is  divided  into  four  parts:  1)  a general 
summary  of  the  objectives,  procedures,  and  findings  of 
this  research;  2)  an  examination  of  the  limitations;  3) 
the  implications  of  this  research  for  German  teachers; 
and  4)  suggestions  for  futher  research. 

Summary 

Obj  ective 

Of  the  four  language  skills  (speaking,  listening, 
reading,  and  writing),  many  educators  consider  reading 
ultimately  to  be  the  most  important  because  so  much  in 
the  foreign  language  classroom  centers  around  the  student's 
ability  to  read.  Despite  this  fact,  however,  teachers 
often  unknowingly  select  reading  materials  far  too  dif- 
ficult for  their  students'  capabilities,  primarily  be- 
cause the  various  measurement  procedures  presently  avail- 
able f oi*  determining  the  difficulty  level  of  a reading 
selection  are  complicated,  time  consuming,  and  have  all 
been  shown  to  be  generally  subjective  and  unreliable. 

The  overall  goal  of  this  research,  therefore,  was  to 
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develop  a simple,  objective,  valid,  and  reliable  testing 
procedure  to  assist  German  teachers  in  choosing  appropri- 
ately difficult  reading  selections  for  their  students. 
Procedure 

To  best  accomplish  the  objective,  it  was  decided  to 
establish  the  relationship  between  the  cloze  test,  a very 
simple  device  to  construct,  and  information  gain,  as 
measured  by  a specially  prepared,  operationally  replicable, 
completion- type  objective  test,  and  thereby  establish  a 
cloze  criterion  score  by  determining  what  percentage  on 
German  cloze  tests,  in  general,  represents  that  point  at 
which  nonnative  readers  begin  to  gain  substantial  infor- 
mation. To  determine  the  relationship  between  the  two 
test  types,  and  thereby  establish  the  cloze  criterion 
score,  both  a cloze  test  and  an  operationally  replicable 
completion  test  were  constructed  from  the  same  German 
reading  passage.  In  both  instances,  specific  guide- 
lines were  laid  out  for  test  construction.  210  college- 
aged,  nonnative  students  of  German,  who  previously  had 
been  divided  into  105  pairs  matched  according  to  German 
reading  ability,  were  then  selected  and  administered  either 
the  cloze  test  or  the  completion  test  so  that  each  matched 
pair  was  represented  by  two  scores,  a cloze  score  and  a 
completion- test  score.  After  scoring  the  two  types  of 
tests  according  to  specific,  predetermined  sets  of  rules, 
oil  findings  were  submitted  to  the  computer  for  a re- 
gression analysis.  In  order  to  find  the  regression  model 
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that  would  best  fit  the  data  and  account  for  the  largest 
amount  of  variance,  several  different  types  of  linear  and 
curvilinear  regression  models  were  experimented  with. 
Ultimately,  the  data  were  fitted  using  a curve  generated 
by  a fifth-degree  polynomial  regression  model. 

Findings 

In  analyzing  the  results  of  the  regression  analysis, 
the  two  types  of  tests  were  found  to  be  highly  related. 
The  multiplecorrelation  between  the  five  powers  of  the 
independent  variable  (cloze  test)  and  the  dependent  vari- 
able (completion  test)  was  .75  with  the  various  powers  of 
the  cloze  test  accounting  for  nearly  56  percent  of  the 
variance  found  in  the  completion  test.  Because  the  re- 
lationship between  the  two  tests  was  curvilinear,  a 
phenomenon  that  had  been  expected  based  on  previous  cloze 
experimentation  conducted  in  English  by  Bormuth,  it  was 
possible  to  establish  a cloze  criterion  score  that  did, 
in  fact,  identify  that  cloze  percentage  point  at  which 
significant  information  began  to  be  won  as  measured  by 
the  completion  test.  As  a result,  it  was  mathematically 
determined  that  40  percent  on  the  cloze  test  best  de- 
scribed that  level  at  which  nonnative  readers  of  German 
would  begin  to  gain  substantial  information  from  reading 
German  prose. 

Limitations 

In  considering  the  findings  of  this  research,  the 
following  two  limitations  should  be  considered: 
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The  Sample 

Because  the  sample  of  German  students  used  in  this 
study  was  limited  to  two  central  Ohio  institutions  of  higher 
learning  that  had  not  been  randomly  selected  from  the  larger 
population  of  all  institutions  of  higher  learning,  and 
because  the  sample  drawn  from  this  limited  population  was 
not  done  so  randomly,  there  exists  the  possibility  that 
the  reported  findings  are  not  completely  general izable 
to  the  larger  population  of  all  col lege- aged , nonnative 
students  of  German.  No  specific  indication  of  such  a 
circumstance  was  detected,  however. 

The  Test-Writing  Algorithm 

A basic  premise  of  this  research  is  that  in  order  for 
the  derived  cloze  criterion  score  to  be  considered  valid 
and  generalizable,  not  only  the  cloze  test,  but  also  the 
objective  test  used  to  measure  information  gain  must  be 
operationally  replicable.  Otherwise,  a variety  of  uncontrol- 
lable test-writer  biases  are  introduced  and  something 
other  than  the  actual  difficulty  level  of  the  passage 
is  measured,  a circumstance  that  proportionately  invali- 
dates the  findings.  Thus,  the  need  for  the  development  of 
a test-writing  algorithm  to  ensure  objectivity. 

Although  the  algorithm  used  in  this  research  docs, 
in  fact,  highly  control  the  test-writing  procedure,  there 
are  nevertheless  two  possible  areas  of  concern:  a) 

According  to  Rule  24,  only  case  ((')  and  scntcncc(S)  consti- 


tuents may  he  replaced  with  an  interrogative  form  to 
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Assuming  that  the  findings  of  this  research  are  valid 
and  generalizable  to  all  cloze  tests  constructed  and  scored 
according  to  guidelines  laid  out  in  Chapter  III,  there 
now  exists,  for  the  first  time,  a very  simple  procedure 
whereby  teachers  of  college-aged,  nonnative  students  of 
German  can  be  assisted  objectively  in  determining  whether  or 
not  the  reading  materials  they  have  selected  for  their 
students  are,  in  fact,  of  the  proper  difficulty  level. 

By  choosing  interesting  reading  material  that  has  been 
objectively  examined  for  difficulty,  a serious  source  of 
frustration  will  have  begun  to  be  eliminated  from  the  foreign 
language  classroom. 

Recommendations 

The  entire  area  of  objectively  assessing  the  appropri- 
ateness of  foreign  language  reading  materials  with  the 
cloze  test  is  completely  virgin  territory.  Consequently, 
there  exists  a host  of  varied  topics  to  be  investigated. 
Several  that  seem  to  be  most  directly  related  to  this  project 
include  the  following: 

1)  The  findings  of  this  research  should  be  confirmed 
by  replication  using  a sample  that  has  been  randomly  drawn 
from  a larger  and  therefore  more  representative  population 
of  colleges  and  universities. 

2)  Because  the  cloze  criterion  score  established  in 
this  research  can  only  be  applied  to  college-aged  students 
of  German,  there  is  a definite  need  to  examine  the  topic 
of  reading  difficulty  at  the  FLHS,  junior,  and  senior 
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high  school  levels  where  findings  may  he  different  because 
native- language  reading  skills  often  have  not  yet  been 
fully  developed. 

3)  To  improve  the  validity  of  findings,  additional 
research  is  needed  in  generating  unbiased,  operationally 
replicable  objective  tests  with  which  to  compare  the  cloze 
test. 

4)  This  research  needs  to  be  replicated  at  all  levels 

in  other  foreign  languages.  Of  special  interest  are  languages 
such  as  Arabic,  Chinese,  and  Russian  that  have  completely 
different  writing  systems. 

5)  Finally,  the  appropriateness  of  a reading  selection 
is  ultimately  not  solely  dependent  upon  whether  or  not 
information  is  being  gained.  Other  factors,  such  as  moti- 
vation and  style,  are  also  important.  Research  should 

be  conducted  to  identify,  assess,  and  incorporate  these 
other  factors  into  the  cloze  criterion  score. 
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Die  Riesen  und  die  Zwerge 

Auf  einem  Schloss,  hoch  oben  auf  einem  Berg,  wohnte 
vor  vielen  Jahren  ein  Ritter  mit  seiner  jungen  Tochter. 

Der  Ritter  und  seine  Tochter  waren  beide  Riesen.  Deshalb 
waren  beide,  Vatcr  und  Kind,  natllrlich  schr  gross  und  schr 
stark.  Trotz  ihrcr  grossed  Gestalt  war  die  Tochter  noch 
ein  gutes  Kind.  Den  ganzen  Tag  spielte  sic  gern. 

Einst,  an  einem  schdnen  Tag  im  Frllhling,  ging  die 
Tochter  spazieren.  Wie  immer,  hatte  sie  es  im  Sinne,  einen 
Spaziergang  auf  die  nahe  Spitze  des  Berges  zu  machen. 

Aber  da  war  sie  schon  so  oft  gewesen.  Weil  sie  einmal 
etwas  Neues  sehen  wollte,  ging  sie  ins  Tal,  wo  sie  noch 
nie  gewesen  war.  Mit  einigen  grossen  Riesenschritten  war 
sie  unten  am  Berge  und  auf  dem  Pelde  im  Tal , wo  die  Bauern 
pflllgten.  Da  stiess  sie  mit  ihrein  Fuss  gegen  etwas.  Von 
ihrer  grossen  Hdhe  sah  sie  auf  den  Boden  hinunter.  Vor 
Freude  liber  das,  was  sie  sah,  klatschte  sie  in  die  HUnde. 

"Wie  schdn!"  rief  sie  frdhlich.  "Das  muss  ich  meinem 
Vater  zeigen!"  Schnell  kniete  sie  nieder,  und  mit  der 
Hand  fegte  sie  alles,  was  sie  vor  sich  sah,  in  ihre  grosse 
blaue  Schllrze.  Frdhlich  lief  sie  nach  Hause.  Vor  Freude 
sprang  sie  leicht  Uber  den  letzten  steilen  Felsen  vor  ihrem 
Schloss.  "Vater,  ach  Vater,  ich  habe  ein  wunderschdnes 
Spielzeug  gefunden,"  rief  sie  und  lief  an  den  Tisch,  wo  ihr 
Vater  beim  Mittagsessen  sass.  "Ei,  das  muss  etwas  Schdnes 
sein,"  sagte  er.  "Deine  Augen  leuchten  ja  vor  Freude." 

"Sieh  in  meine  Schllrze!  Kannst  du  sehen,  was  es  ist?" 
fragte  das  MUdchen.  "Es  ist  zu  dunkel  in  der  Schllrze. 

Ich  kann  nichts  sehen,"  sagte  der  Vater  und  lUchelte. 

"Aber  ich  fllhle,  wie  das  Ding  sich  bewegt."  Die  Tochter 
dffnete  nun  ihre  Schllrze  und  stellte  vorsichtig  alles,  was 
sie  auf  dem  Felde  gefunden  hatte,  vor  dem  Vater  auf: 
einen  Pflug,  zwei  Pferde  und  einen  Bauern  stellte  sie  auf 
den  Tisch.  Dann  klatschte  sie  in  die  HUnde  und  lachte 
laut  liber  den  komischen  Zwerg,  den  Bauern,  der  voll  Angst 
auf  dem  Tische  umherlief.  Auch  der  Vatcr  lUchelte  Uber 
den  komischen  Anblick.  Dann  aber  sagte  er  ernst  zu  seiner 
Tochter:  "Mein  Kind,  das  ist  kein  Spielzeug;  gehe  gleich 

wieder  hinunter  und  bringe  alles  aufs  Feld  zurllck."  "Aber 
Vater,  warum  denn?"  fragte  das  Mfidchcn  ganz  ungltlcklich 
als  sie  den  Bauern  und  die  Pferde  in  ihre  IIHndc  nahm. 

"Solch  ein  schdnes  lebendiges  Spielzeig  habe  ich  noch  nie 
gchabtl"  "Der  Bauer  ist  kein  Spielzeug,  sage  ich  dir," 
sagte  der  Vater.  "Er  muss  unten  in  dem  Tal  fUr  uns  arbelten. 
Er  muss  pflUgen,  sMen  und  ernten;  sonst  haben  wir  Riesen 
oben  auf  unscrcn  Schldsscrn  nichts  zu  essen  und  mUssen 
hungern.  Also  trage  ihn  schnell  dahin,  wo  du  ihn  gefunden 
hast."  Die  Tochter  weinte,  aber  gehorsam  packte  sic  den 
Pflug,  die  Pferde  und  den  Bauern  wieder  in  ihre  Schllrze 
und  hrachte  sic  auf  das  Feld  zurllck.  "Wie  schadc," 
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seufzte  sie  dann  und  trocknete  ihre  TrHnen  mit  ihrer 
blauen  Schlirze.  "Wie  schade,  dass  mein  Vater  so  streng 
ist . " 
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Das  Kamel 

Einst  schrieb  eine  berllhmte  UniversitHt  einen  Preis 
aus  fUr  die  beste  schr  i f 1 1 iche  Arbe  i t liber  das  Kamel. 

Unter  den  Kandidatcn,  die  den  Preis  gewinncn  wollten,  waren 
ein  Amerikaner,  cin  EnglMndor,  cin  Franzose  und  ein  Deutscher. 
Jedcr  von  ihncn  schrieb  einen  Aufsatz  liber  das  Kamel,  und 
jeder  verfolgte  dabei  eine  anderc  Methode. 

Der  praktische  Englander,  der  seinen  Augen  mehr  traute 
als  seinem  Gehirn,  packte  seine  Sachen  und  ging  mit  Bade- 
wanne  und  Teetasse  in  die  WUste  Sahara.  Dort  lebte  er  auf 
einer  Oase  ganz  wie  in  England.  Jcden  Tag  badete  er  fleissig 
und  trank  viel  Tee.  Daneben  spielte  er  Tennis  und  Cricket, 
wahrend  seiner  Freizeit  studierte  er  das  Kamel  in  der 
WUste.  Nach  fUnf  Wochen  hatte  er  genug  Material  fUr  den 
Aufsatz  und  ging  deshalb  wieder  nach  England  zurUck.  Dort 
schrieb  er  ohne  grosse  Problcme  und  in  klarcn  kurzen  Worten 
alles,  was  er  in  der  Heimat  der  Kamele  geschen  hatte. 

In  der  Zwischenzeit  hatte  der  Franzose  seine  Arbeit 
schon  langst  beendet.  Er  war  in  Paris  geblieben  und  hatte 
einige  Stunden  vor  den  Kamelen  im  Zoo  verbracht.  Die  Tiere 
mit  ihrer  phantastischen  Gestalt  hatten  ihn  sehr  amUsiert. 
Sofort  hatte  er  eine  lustige  Plauderei  in  seinem  Kopf 
Uber  das  Kamel  skizziert.  Schnell  war  er  in  ein  nahes 
Cafe  gegangen,  urn  die  Skizze  auszuarbeiten . Diese  war  voll 
lustiger  und  dramatischer  Situationen.  Wissenschaf tl ich 
war  sie  aber  nicht. 

Der  Amerikaner  hatte  lange  keine  Zeit  finden  kbnnen, 
den  Aufsatz  zu  schreiben.  Er  ist  immer  so  furchtbar 
beschHftigt.  Er  ist  Mitglied  von  zehn  Klubs  und  Vor- 
sitzender  von  zehn  anderen.  Endlich  kommt  der  Tag,  den  er 
fUr  die  Fabrikation  des  Aufsatzes  reserviert  hatte.  In 
die  WUste  kann  er  nicht  gut  gehen.  Den  Zoo  kann  er  auch 
nicht  besuchen,  weil  er  in  Boston  wohnt.  Wo  findet  er  nun 
so  schnell  wie  mUglich  das  nbtige  Material  fUr  seinen 
Aufsatz?  Nur  in  der  "Public  Library"  bekommt  er  die 
Information,  die  er  braucht.  Dort  kann  er  alles,  was  je 
Uber  das  Kamel  geschrieben  worden  ist,  schnell  Uber- 
fliegen.  Am  nHchsten  Abend  kommt  er  mit  einem  vollen 
Notizbuch  nach  Hause.  Er  setzt  sich  in  den  Schaukelstuhl 
und  arrangiert  sein  Material.  Sehr  schnell  stellt  er  mit 
Hilfe  seiner  Notizen  eine  Bcschreibung  des  Kamels  zusammen. 

Der  Deutsche  hat  viel  mehr  als  alle  andern  zusammen 
geschrieben.  Er  hat  sogar  einen  ganz  dickcn  Band  geschrieben. 
An  demselben  Tag,  an  dem  er  das  Prei sausschrcibcn  der 
UniversitHt  gclcscn  hatte,  schloss  er  sich  in  sein  Studicr- 
zimmer  cin.  Niemand  ausser  seiner  Frau  durftc  hcrcin- 
kommen.  Dort  sass  er  im  langcn  Schlafrock,  mit  einer  noch 
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longeron  Pfcife  i in  Mundc,  und  dachtc  lag  und  Nuclit  Ubcr  das 
Kamel.  Von  Zeit  zu  Zeit  kam  seine  Frau  auf  den  Zehen- 
spitzen  ins  Zimmer,  urn  ihm  sein  Essen  zu  bringen.  Diese 
strenge  Arbeit  dauerte  viele  Wochen.  Der  arme  Mann  ass 
immcr  weniger  und  rauchte  immcr  mehr.  Nach  zwei  Monaten 
sail  er  wie  cin  Gespcnst  aus . Wenn  man  ihn  sail,  sass  er 
immcr  in  cincr  Raucliwolkc  vor  seinem  Schrc  ibt  i sch . Mit 
gekrUmmtem  RUcken  und  gcbeugtem  Kopl'e  konnte  man  ihn  wirk- 
lich  kaum  von  einem  Kamel  unterscheiden . Langsam  ging 
die  Arbeit  voran,  denn  der  Gelehrte  hatte  es  nicht  leicht. 
Weil  ein  Deutscher  alles  immer  grUndlich  macht , musste 
er  das  Kamel  aus  den  Tiefen  seiner  cigenen  Seele  entwicklen. 
Und  eine  deutsche  Seele  ist  sehr  tief.  Nach  drei  Monaten 
war  das  grosse  Werk  fertig,  und  dcr  Gelehrte  stiess  einen 
sehr  grossen  Seufzer  dcr  Erlcichterung  aus.  Mit  triumph- 
ierendcr  Freude  schrieb  er  auf  das  Titelblatt  des  dicken 
Manuskriptes : "Das  ideale  Kamel,  eine  Philosophic  des 
Unbewussten . " 

Wer,  meinen  Sie  wohl,  bekam  den  Preis? 
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Name 

(Pieane  Prim  ) 

Instructor 

Course  Humber 


INTRODUCTION 


Nothing  is  more  frustrating  for  you,  as  a rtuaent,  than  to 
be  required  to  read  foreign  language  material  with  e difficulty 
level  far  above  your  own  language  capabilites— a phenomenon  that 
car:  easily  occur  ir.  the  foreign  language  classroom.  Ir.  order  to 
solve  this  problem,  you  have  been  selected  to  participate  in  an 
educational  research  study  designed  to  provide  German  instructors 
with  empirical  data  that  will  enable  them  to  accurately  choose 
appropriate  student  reading  selections. 

You  will  be  administered  two  snort  tests.  Test  I is  to  be 
administered  today.  Test  II  will  be  administered  in  the  very 
near  future,  please  do  your  very  best.  The  more  conscientiously 
you  participate,  the  more  meaningful  the  findings  will  be  to 
both  instructors  and  future  students  of  German. 


INSTRUCTIONS 

The  attached  test  has  beer,  made  by  simply  deleting  every 
eighth  word  from  a short  story.  Your  task  is  to  first  read 
through  the  story  to  see  what  it  is  about  and  then  to  try  to 
fill  in  each  blank  with  the  German  word  that  makes  most  sense 
to  you.  You  are  to  write  only  one  word  in  each  blank.  If  you 
are  not  sure  of  the  word,  do  not  be  afraid  to  guess.  Try  to  write 
an  answer  for  every  blank.  Some  blanks  will  be  quite  easy  and 
some  will  be  very  difficult.  To  be  most  effective,  do  not  sperd 
too  much  time  on  any  one  blank.  The  answers  to  some  items  will 
be  Immediately  apparent.  By  completing  all  of  these  items  first 
and  thereby  providing  additional  context  and  meaning  to  the 
passage,  you  may  increase  your  chances  for  answering  tome  of  the 
items  you  find  to  be  more  difficult.  All  of  the  blanks  can  be 
answered  with  ordinary  German  words.  Wrong  spelling  will  not  be 
counted  incorrect  if  the  test  grader  can  tell  what  word  you 
intended.  Please  write  neatly  and  remember,  dc  NOT  write  more 
than  one  word  in  each  blank.  You  should  be  able  to  complete  this 
test  in  about  20-30  minutes.  The  test  is  not  timed,  however. 

When  you  are  finished,  raise  your  hand  and  your- teat  material 
will  be  collected.  If  you  have  any  questions,  pl-caue  ask  then 
at  this  time. 


(Do  not  tupn  the-  page 
until  told  to  dc  so . ) 
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TEST  I 


Auf  eiriem  Schloli,  hoch  oi>en  auf  einen  Ber-fi,  wohnte  vor 
vlelen  Jahren  ein  F.ltter  nit  seiner  jungen  Tochter.  Iier  Ritter 

und  Tochter  waren  beide  hiesen.  DeGhalb  waren  beide, 

und  Kind,  natUrlich  senr  groB  und  sehr  . 

Trotz  ihrer  fir ofien  Gestalt  war  die  Tochter  em  gutefe 

Kind.  Den  ganzen  Tag  spielte  gern. 

Einst,  an  einen  scnonen  Tag  in  , cing  die  Tochter 

spazieren.  Vile  loner,  hatte  es  in  Sir.ne,  einen 

Spaziergang  auf  die  Spltze  dee  serges  zu  nacner. . 

Aber  da  sle  schon  so  oft  gewesen.  Well  sie  

etwas  Keues  sehen  wollte,  glng  oie  ins  , wo  sie  noch 

nle  gewesen  war.  Hit  grofien  Riosenschritter.  war  sie 

unten  an  Berge  auf  den  Beide  in  Tal,  wo  die  

pflligten.  Da  stieG  oie  mit  ihreo  FuG  etwas.  von 

ihrer  groBen  Htihe  oah  sie  oen  Boder.  hlnunter.  Vor 

Freude  liber  daB,  sie  sah , klatschte  sie  in  die 

Hands.  " schon!"  rief  sie  frol.lich.  "Das  wuB  ich 

Vater  zeigen!"  Schnell  kniete  sie  nieder,  und 

dor  Hand  fegte  sie  alles.  was  sie  slch 

sah,  in  ihre  gro&e  blaue  SchUrze. lief  sie  nach 

Hause.  Vor  Freude  sprang  lelcht  liber  den  lctzten 

stellen  Felsen  vor  SchioB.  "Veter,  ach  Vbter,  ich 

habe  ein  Splelzeug  gefunden,"  rief  sie  und  lief  an 

Tisch,  wo  ihr  Vater  bein  Kittagsessen  oat. 

" , das  muB  etwae  Schdnes  sein,"  uagte  er.  " 

Auger,  leuchten  Ja  vor  Freude."  "Sier.  in  ScnUrzel 

Kannst  du  sehen,  was  es  lot?"  das  f'.adchen.  "Do  1st 

zu  dunkel  in  ________  SchUrze.  Ich  kunn  nlchtc  ocnon," 

cogte  der  und  lachelto.  "Aber  ich  fUhle  wie  das 

_________  eich  bowegt."  Die  Tochter  bffnetc  nur,  ihre 

und  stellte  vorslchtig  alles,  wuo  sie  auf  Feloe 

gefunden  hatte,  vor  den  Vater  aufi  Pflug,  zwei 

Pferde  und  einen  Bauern  stellte  auf  dei.  Tlnch. 

Dann  klntschte  sle  ir.  Hnnoc  unr.  lachtc  iu,t  Uber 
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den  komlschen  , den  Bauern,  der  voll  Angst  auf  dec 

umherlief.  Auch  der  Vater  lachelte  liber  den  

Anblick.  Dann  aber  augte  er  ernet  zu  Tochter: 

"Hein  Kind,  dae  iat  kein  Spielzeug;  gleich  wieder 

hinunter  und  brlnge  alles  aufa  zurtick."  "Aber  Vater, 

warum  denn?"  fragte  dae  ganz  uneliicklich  ale  eie  den 

Bauern  und  Pferde  in  ihre  Hanae  nahiL.  "Solck  eir. 

lebenaigee  Spielzeug  habe  ich  noch  nie  gehabt!" 

" Bauer  iat  kein  Spielzeug,  sage  ich  dir,"  

der  Vater.  "Er  nuB  unten  in  den  fur  une  arbelten. 

Er  muB  pflUgen,  eaen  erntenj  eonst  haben  wir  Kieaen 

oben  auf  Schlbesern  nichts  zu  eeeen  und  nusaen 

hungern.  trage  ihn  echnell  dahin,  wo  du  ihn  

hast."  Die  Tochter  weinte,  aber  gehorsam  packte  den 

Pflug,  die  Pferde  und  den  Bauern  in  ihre  Schtirze 

und  brachte  ale  auf  Feld  zurtick.  "Wie  ochade," 

aeufzte  eie  dann  trocknete  ihre  Tranen  mit  ihrer 

blauen  Schtirze.  "Wie  schade,  daB  mein  Vater  eo  atreng  let!" 
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msraucTioiis 

The  attached  test  has  been  constructed  in  a manner  similar 
to  the  last  test  yo u were  administered,  i.e.,  ever/  eighth  word 
has  simply  been  deleted  fron  a short  story.  Your  task  is  to 
first  read  through  the  story  to  see  what  it  is  about  ana  then  to 
try  to  fill  in  each  blank  witu  the  Oernan  word  that  nukes  .most 
sense  to  you.  You  are  to  write  only  one  word  in  each  blank.  If 
you  are  not  sure  of  the  word,  Jo  not  be  afraid  to  guess.  Try 
to  write  an  answer  for  evory  blank,  dome  tlar.ks  will  be  'iuite 
easy  and  some  will  be  very  difficult.  To  be  moot  effective, 
do  not  spend  too  much  tiae  on  any  one  blank.  The  answers  to  some 
items  will  be  immediately  apparent.  By  completing  all  of  thece 
items  first  and  thereby  providing  additional  context  and  meaning 
to  the  passage,  you  may  increase  your  chances  for  answering  some 
of  the  iter.s  you  find  to  be  more  difficult.  All  of  the  blanks 
can  be  answered  with  ordinary  Gernan  words.  Wrong  spelling  will 
not  be  counted  incorrect  If  the  test  grader  can  tall  what  word 
you  intended.  Please  write  neatly  and  remember,  do  MOT  write 
more  than  one  word  in  each  blank.  If  you  have  any  nutations, 
please  raise  your  hand.  Otherwise,  you  nay  begin  work  now. 
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( Form  I l 


Einat  schrieb  elne  berUhmte  Univeraitat  einen  Prels  bus  fllr 
die  beste  schrif tllche  Arbeit  liber  due  Kamel.  Ur.ter 

Kund’idaten,  die  den  Prels  gewinnen  wollten,  waren. 

Asierlkaner , ein  Engl&nder,  eir.  Franzose  und  ein  . 

Jeder  von  ihnen  schrieb  einen  Aufsatz  liber  Kamel, 

und  Jeder  verfolgte  dabei  eine  andere  . , 

Der  praktische  Englander,  der  aeinen  Augen  mehr  

als  seinem  Gehirn,  packte  seine  Sachen  und  nit 

Badewanne  und  Teetaeae  in  die  Wuste  . Dort  lebte  er 

auf  einer  Oase  ganz  in  England.  Jeder.  Tag  badete  er 

fleiBig  trank  viel  Tec.  Daneben  spielte  er  Tennis 

Cricket.  Whhrend  seiner  Freizelt  studierte  er  das 

in  der  WUste.  Uach  flinf  Wochen  hatte  

genug  Material  fUr  den  Aufsatz  und  gine  wieder  nach 

England  zurlick.  Dort  schrieb  er  grobe  Probleoe  und 

in  klaren  kurzen  Worten  , was  er  in  der  Heimfit  der 

Kaoele  hatte. 

In  der  Zwiscbenzelt  hatte  der  Franzoee  Arbeit 

achon  ltngat  beendet.  Er  war  in  ; geblieben  und  hatte 

elnige  Stunden  vor  den  in  Zoc  vertracht.  Die  Tiere 

nit  lhrer  Gestalt  hatter  ihn  schr  amUsiert.  Sofort 

hatte  eine  lustige  Plauderei  in  seiner.  Kopf  hber 

F.amel  sklzziert.  Schr.ell  war  er  in  ein  

Cafe' gep.angen,  ur.  die  Skizze  auazuurbeiten.  Dies**  

voll  lustlger  und  dramatlscher  Situatior.cn.  Kiaccnachuf tlich 
war  uber  nlcht. 

Der  Auerikaner  hatte  lange  keinc  flnden  kttnnen, 

den  Aufeatz  zu  echreiben.  Er  immer  eo  furchtbar 

beachaftigt.  Er  iat  Mltglied  zehn  Klub*  und 

Voreitzender  von  zehn  anderen.  ______  kommt  der  Tag,  den 

er  fUr  die  dee  Aufsatzes  reaervlert  hatte.  Ir.  die 

WUate  er  nlcht  gut  gehen.  Den  Zoo  kanr.  

auch  nlcht  beauchen,  well  er  in  Doctor  . Wo  flndet  er 

nun  ao  echnell  wie  _______  das  nbtlge  Material  fUr  aelner. 

Aufaatz?  Kur  der  "Public  Library"  bekomc.t  er  die 
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Information , 


er  braucht.  Dort  kann  *r  allec,  was 


Uber  (ins  Kanel  geschrieben  worden  iet,  ochnell 

Am  nSchbten  Abend  komret  er  r.it  einem  

Er  setzt  sich  in  


Notizbuch  nach  Hauae . 

und  arrangiert  sein  Material.  Sehr  ochnell 

Eilfe  seiner  Rotizen  eine  Beschreibung  

Der  Deutsche  hat  viel  mehr  

geschrieben.  Er  hat  sogar 


Schaukcl e tuhl 
er  mlt 


Kamels  zusamraen. 


bile  bndern  zusammen 


An  denselben  Tag,  

Unlversitlit  gelesen 
zinmer  ein.  _____ 

Dort  saB  

lJingeren  


genz  dicker.  Band  geachrieben. 

dec  er  dae  Preisaueschreiben  der 
, scliloB  er  sich  in  sein  Studier- 


aufler  seiner  Prau  durfte  hereinkonmen. 


iD  langen  Schlafrock,  Bit  einer  noch 
iB  Munde,  urid  dachte  Tag  und  Nacht  


das  Kamel.  Von  Zeit  zu  Zeit  kan 

Zehenspitzen  ins  Zinroer,  urn  

Dieee  strenge  Arbeit  

aB  inmer  und  rauchte  iwoer  nehr. 


Prau  auf  den 


wirklich  kaum  von  einem 
die  Arbeit  voran,  denn 
Well  elr.  


sein  Essen  zu  bringen. 

viele  Wochen.  Der  arme  Mann 

Kach  zwei  Mona  ten 

er  wie  ein  Gespenet  aus.  Wenn  man  Bah, 

eeinem  Dchreibtiech. 

________  korrnte  man  ihn 

unterscheiaen . Dangsam  ging 
Gelehrte  hatte  es  nicht  leicht. 


safe  er  imuer  in  einer  Kauchwoike  

Kit  gekrUmmtem  RUeken  und  gebeugtem 


alles  iiraier  grlindlich  maoht,  mufite  er  das 


aus  den  Tiefen  seiner  eigenen  Seele  entwickeln. 
eine  deutuche  Seele  lot  sehr  tief.  bach  


Monaten  war  das  grofle  Work  fertig,  und 
stieB  einen  sehr  gro6en  Seufzer  der 


triumphierender  Preude  schrieb  er  auf  

des  dicken  Manuskriptea s "Das  Ideal e Kanel, 
Philosophic  des  UnbewuBten." 

Ker,  melnen  Sle  wohl,  bekam  den  Preis? 


Gelehrte 

aus.  Kit 
Ti telblatt 


< 


1 


1 


I 

I I 


I 
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TEST  IU 


(Torn,  ?) 


Einat  achrieb  elne  berUhmte  Univeraitkt  elnen  Praia  aua  fur 
die  beete  schriftliche  Arbeit  Uber  das  Kamel.  Unter  aer.  Kanai- 

daten,  die  Preis  gewlnnen  wollten,  waren  ein 

Anerikener,  ein  , eir.  Pranzose  und  ein  Deutacher. 

Jeder  von  ________  achrieb  einer,  Aufsatz  Uber  das  Kamel, 

und  verfolgte  dabei  eine  aridere  Xethode. 

Der  praktiache  , der  seiner  Au£en  r.iehr  traute 

ala  eeinem  , packte  seine  Cachen  und  ging  mit  Haaewanne 

Teetaeae  in  die  Wuete  Sahara.  Dort  lebte  

auf  einer  Oaee  pan z wie  in  England.  Tag  badete  er 

fleiBig  und  trank  viel  . Daneben  spielte  er  Tenr.ie 

und  Cricket,  wahrend  Preizeit  studierte  er  das 

Kamel  in  der  . Kech  fUnf  Uochen  hatte  er  genug 

Material  den  Aufsatz  und  ging  deshalb  wieder  nach 

_________  zurUck.  Dort  achrieb  er  ohne  grolje  Probleme 

in  klaijen  kurzen  Worten  alles,  was  er  

der  Helmet  der  Kamele  geaehen  hatte. 

In  Zwiachenzelt  hatte  der  Pranzooe  seine  Arbeit 

achon  beendet.  Er  war  in  PariB  geblieben  und 

einige  Stunden  vor  den  Kamelen  in  Zoo  . 

Die  Tiere  mit  ihrer  phantaatlschen  Geotalt  batten  

aehr  antialert.  Sofort  hatte  er  elne  luatige  in 

aeinem  Kopf  Uber  das  Kamel  ekizziert.  war  er  in  ein 

nahes  Cafe'  gegangen,  die  Skizze  auezuortei ten . 

Dleee  war  voll  luatiger  ________  dramatiacher  Sltuationen. 

Wiseenschaftlich  war  ale  aber  r.icht. 

Anerikaner  liatte  lange  kelne  Zelt  finden  kdnnen, 

Aufoatz  zu  achreibcn.  Er  1st  inner  eo  

beechfeftigt.  Er  iat  Mltglled  von  zehn  Kluba  Voraltzender 

von  zehn  anderen.  Endllch  kommt  der  , den  er  fUr 

die  Fabriketlon  dee  Aufsatzes  hatte.  In  die  WUcte 

kann  er  nicht  gcher. . Den  Zoo  kanrj  er  such  r.icht 

, well  er  in  Boeton  wohnt.  Wo  fir.det 

nun  eo  sehneil  wie  mbelich  dae  nbtlge  _______  fUr  seiner. 

Aufsatz?  Kur  ir.  der  "Public  " bekoitmt  er  die 


bekoitmt  er  die 
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Information,  die  er  braucht.  kanr.  er  alles,  was  Je 

liber  das  geechrleben  worden  ist,  achnell  Uberfliegen. 

Am  nachsten  

nach  


kom^t  er  sit  einem  vcllen  Jlotizbucl. 


. Er  cetzt  sich  in  den  Schaukelstuhl  und 
sein  Material.  Sehr  achnell  stellt  er  Bit  


seiner  Kctizen  eine  Beachreitunc  des  Kariele  zusammen. 

Deutsche  hat  vtel  mehr  als  alle  ar.derr. 


geschrleben. 
f eschrieben . 


Er  hat  sogar  einen  ganz  dicken 
An  demselben  Tag,  an  dem  er  


Prele- 


r.it  einer 


auaechreiben  der  Dnivereittit  geleser.  hatte,  ichlct  er 

in  sein  Studierzlnmer  ein.  Niemanc  aufcer  seiner  

durfte  hereinkommen.  Dort  saG  er  1m  langen  

noch  langeren  Pfelfe  im  Munde,  dachte  Tag  und  1/acht 

liber  das  Kamel.  _________  Zeit  zu  Zeit  kam  Beine  Frau  auf 

_________  Zehenspltzer.  ins  Zimmer,  urn  ihm  sein  Esser,  


bringen.  Diese  etrenge  Arbeit  dauerte  viele  Wochen.  

arne  Mann  aE  lmmer  weniger  und  rauchte  r.ehr. 

zwei  Monat.en  eah  er  wie  Gespenst  uut  . 

eah,  8t3  


Each 
i t nr.  man  :hr. 


Sehreibtisch . 


Kopfe  konnte  man  

ur.terscheider. . Dangsan 

Gelehrte  hatte  

alles  lmraer  


iramer  ir.  einer  Kauchwolke  vor  seiner. 

gekrlimmtejr.  Rlicken  und  gebeugtem 

wirklich  kaum  von  einer.  Kamel 

die  Arbeit  vcran,  denr  der 
Weil  ein  Deutscher 


nicht  leicht. 


macht,  aiul'te  er  dan  Kamel  aue  den 


seiner  elgenen  Deelo  entwickeln.  Und  eine  Jrutrche 
1st  sehr  tief.  Mach  drei  lionaten  war  


groBe  Werk  fertig,  und  der  Gelehrte  aticfc 
groBen  Seufzer  der  Erleichterung  aur.  Mil 
schrleb  er  auf  das  Titelblatt  des  


»ehr 

Preude 


lianuekr  iptes : 


"lias  ideal  e Kamel,  eine  philosophic  dco 


V/er,  rneinen  Sie  wohl,  bekam  den  Prels? 
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TEST  IIA 


( lorn,  .'I 


Einst  schrieb  eine  berlihmte  Universitat  eirien  Preif  aus  fur 
die  beote  schriftliche  Arbeit  liber  dau  Kauiel.  Unter  den  Kandi- 

daten , den  Preia  gewinnen  wollten,  waren  ein 

Awerikaner , Englander,  ein  Franzose  und  ein  Deutscher. 

Jeder  ihnen  schrieb  einen  Aufsatz  Uber  das  Kamel, 

jeder  verfclgte  debei  eine  andere  f'.ethooe. 

Englander,  der  seinen  Auger,  raehr  traute  als 


Der 


Cehirn,  packte  aeine  Eachen  und  ging  nit 


und  Teetas8e  in  die  WUete  Sahara.  Dort  er  auf  ciner 

Oase  ganz  wie  in  . Jeder.  Tag  badete  er  fleifcig  und 

trank  ________  Tee.  De.neben  spielte  er  Tennis  ur.d  Cricket. 


seiner  Freizeit  etudierte  er  das  Kamel  in 


WUete.  Each  flinf  Wocher.  hatte  er  cer.ug 

Aufsatz  und  ging  deshalb  wleder  

schrieb  er  ohrie  grofie 
alles,  waE  


ftir  der. 


England  zurUck.  Dort 


und  in  klarer.  kurzen  Worten 


Arbeit 


in  der  Keirnat  der  Kamele  geoehcn  hatte. 

der  Zwiachenzeit  hatte  der  Franzose  seine 
langst  beendet.  Er  war  in  Paris  geblieben 


hatte  einige  Stundcn  vor  den  Kamelen  in 


verbracht.  Die  Tiere  Dit  ihrer  phantastischer.  Gestalt  

ihr,  sehr  amvisiert.  Sofort  hatte  er  eine  Plauderei 

in  seinen  Kopf  UVer  das  Kamel 

nahes  Cafe'  

war  voll  


Schnell  war  er  in  ein 


, urn  die  Skizze  auszuarbel ten . Diese 

und  drauatischer  Situationen.  V.  isaenschaftlich 


war  sie  aber  . 

Der  Amerika.ner  hatte  iange  keine  Zeit  finder. 

den  Aufsatz  zu  schreiben.  Er  iet  inner  

beschaftigt.  Er  ist  Kitglied  von  zehn  


sitzender  von  zehn  anderen.  Endlich  kommt 


furehtber 
unc  Vor- 
Tag,  den 


er  fiir  die  Pabrikation  den 
WUete  kann  er  


reetrviert  hatte.  In  die 


gut  genen.  Den  Zoo  kann  cr  such 


bcsuchen,  weil  er  ii,  boston  wohnt.  Wo 


er  nui,  or  achnell  win  ndglloh  due 

seiner.  Aufsatz?  I!ur  in  der  " 

Information,  die  cr  . 


Haterjui  fur 


Library"  bekoror.t  er  die 


Dort  kann  er  ulies,  war  Je 


1 

\i 

i * 
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liber 

Am 


Kamel  geschrieben  voider,  ist,  schnell  uberfllegen. 


Abend  komrat  er  nil  eineri  voller.  hoticbuch 


Hause . Hr  setst  sich  in  den  Sc haukelstuhl 

eein  Material.  Sehr  schr.ell  slellt  er  

Notizen  eine  Beschreibung  des  Kamels  

Der  Deutsche  hat  viel  mehr  als  alle  

geschrieben.  Kr  hat  sogar  einen  ganz  

An  demeelben  Tag,  an  dem  


arrar.giert 

Kilfe  seiner 

zusammen 


_ Schlafrock,  nit  ciner  noth 
und  dachte  Tag  und  hacht  uber 


’Jniversitat  geleser.  hatte,  schloB 
Studlerzimmer  ein.  Nienand  auber 

konr.en.  Dort  sab  er  im  

lungeren  Pfeife  im 
das  . Von  Zeit  zu  Zeit  kam  seine  Frau 

den  Zehenapitzen  ins  Zimmer,  um  ihm  oein  

Diese  strenge  Arbeit  dnuerte  viele 

afi  imraer  weniger  und  

sah  er  

er  iramer  in  einer  Raucriwolke  vor  oeinea 


Band  geschrieben. 

das  Preisausechreibcn  der 

sich  in  sein 

Prau  duri'te  herein- 


zu  tringen. 


. Der  arme  Mann 

inmer  mehr.  l.'ach  zwei  fionaten 
ein  Cespenct  aue.  Kenn  man  ihn  sah,  


Kit 


gekrUmntem  Rlicken  und  gebeugtem  Kopfe  konr.te 
wirklich  kaum  von  eineo  Kernel  unterscheiden. 

die  Arbeit  voran,  aenn  der  Gelehrte  

Weil  ein  Deutscher  alles 
das  Kamel  aus  _____ 

Und  eine  


ihn 

. Mine 


es  nicht  leicht. 


grlindlich  rnacht,  muBte  er 


Tiefen  seiner  eigenen  beele  entwickeln. 

Seele  ist  aehr  tief.  l.’ach  drei  Kor.aten 


das  groBe  l.'erk  fertlg,  und  der  Gelehrte 


einen  sehr  groBen  Seufr.er  der  Srleichterung  aus.  

triunphierender  Freude  echrieb  er  auf  das  Titelblatt  

dicker.  Manuskriptes : "Das  iaeale  Kamel,  eine  Phiiosophir 
UnbewuBten." 

Wer,  meinen  Gie  wohl,  bekan  del.  Preis? 


I 


/ 
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IKSTRUCTIOKS 

Attached  is  a short  story  in  German.  You  are  to  reed 
this  story  until  you  feel  you  understand  it  as  well  us  you 
personally  are  capable.  To  achieve  this,  it  is  suggested  that 
you  read  the  story  carefully  through  at  least  two  or  three 
times.  When  you  are  satisfied  that  you  have  achieved  your 
maximum  understanding  level  of  the  story,  raise  your  hand. 

The  8tory.wlll  then  be  collected  and  you  will  be  given  a 38- 
question  completion  test  tc  measure  your  comprehension  of  the 
passage.  The  test  questions  are  ail  short  and  can  be  answered 
in  one  sentence  or  less.  Some  questions  are  rather  general  and 
therefore  quite  easy,  and  some  require  understanding  and  recol- 
lection of  rather  specific  information.  Remember,  you  will 
HOT  have  access  to  the  story  while  you  are  taking  the  test.  It 
is  therefore  extremely  important  that  you  read  the  story  very 
carefully.  If  you  have  any  questions,  please  raise  your  hand. 
Otherwise,  begin  reading  the  attached  story  now. 
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Hame  

(Please  Print) 

Instructor 

Course  Number 


TEST  I IB 


This  test  is  intended  to  measure  you r understanding  of  the 
short  story  you  have  just  read.  It  consists  of  36  short  questions 
that  car.  be  answered  in  one  sentence  or  less.  Some  questions  are 
very  easy, and  some  require  an  understanding  and  recollection  of 
rather  specific  information.  Because  the  test  is  intended  to 
measure  your  understanding  of  the  story,  you  ma/  answer  the 
questions  in  either  German  or  English.  If  possible,  answer 
them  in  German.  You  will  not  be  penalized  for  incorrect  Gram- 
matical constructions  or  spelling.  Nevertheless,  on  any  given 
question  in  which  you  feel  you  can  express  yourself  more  fully 
or  clearly  in  English,  please  do  so.  Remember,  the  objective 
of  the  test  is  to  measure  your  understanding  of  the  story. 

Please  write  clearly  and  neatly.  When  you  have  finished  the 
test,  raise  your  hand  end  your  test  booklet  will  be  collected. 


1 . 

Was  schrieb  einet  eine  beriihmte  Universltat  aus? 

2. 

Wer  schrieb  einen  Aufsatz  iiber  das  Kamel? 

3. 

Wo  studierte  der  Englfinder  das  Kamel  wghrend  seiner  Preizeit? 

4. 

Was  hette  der  Pranzose  in  seinerc  Kopf  sofort  skizziert? 

5. 

Was  alles  kanr.  der  Acerikaner  in  der  "Public  Library"  schnell 
bberflieger.? 

6. 

Wer  dachte  Tar  und  JJacht  iiber  dat  Kamel? 

7. 

Wovon  konnte  man  den  Deutschen,  mit  gekrummtem  Rucker,  und 
gebeugten  Kopfe,  wirklich  kaure  unterscheiden? 

6. 

Wie  muBte  der  Deutsche  das  Kamel  entwickeln,  weil  er  alles 
immer  grtindlich  macht? 

9. 

Wer  schrieb  reit  triumphlerender  Preude  auf  das  Titelblatt  dee 
dicker  Manuskrlptes : Das  ideeie  Kamel,  eine  Philosophic  dee 
DnbewuCten? 
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10.  Wc  lebte  der  Englander  in  der  Wliste  ganz  wie  in  England?. 


11.  Wohin  ginfi  der  Englander  vieder  zurtick,  weil  er  genug  Material 
fur  der.  Aufsatz  hatte? 


12.  Wie  lange  hatte  der  Pranzose  vor  den  Kamelen  im  Zoo  verbracht? 


13.  Wer  kann  den  Zoo  nicht  besuchen,  veil  er  in  Boston  wohnt? 


14.  Wer  badete  jeden  Tag  fleiBig? 


15.  Wohin  ging  der  Englander  cit  Baoewanne  und  l'eetasee?. 


16.  Wer  hatte  in  der  Zwischenzeit  seine  Arbeit  scnon  langst 
beendet? 


17.  Was  epielte  der  Englander  daneben?. 


18.  Wer  auSer  der  Prau  des  Deutschen  durfte  ins  Studierzimner 
hereinkommen? 


1°.  Vomit  stellte  der  Amerikaner  sehr.schnell  eine  Beschreibur.g 
des  Kamels  zueammen? 


20.  Wo  war  der  Pranzose  geblieben? 


21.  An  welchen  Tag  schioE  der  Deutsche  eich  in  sein  Studierziramer 
ein? 


22.  Was  bekommt  der  Amerikaner  nur  in  aer  "Public  Library"? 


23.  Worin  setzt  cich  der  Amerikaner?. 


24.  Karum  kam  die  Prau  des  Deutscher.  auf  den  Zehenspitzen  ins 
Zimmer  vcn  Zeit  zu  Zeit? 


. Vomit  kommt  der  Amerikaner  aa  nsichotcn  Abend  nach  Hause? 
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Explanation 

Included  along  with  each  sentence  analysis  in  Appendix 
D are  the  following  four  items:  1)  the  prepared  (modified) 
sentence  (S)  from  which  the  analysis  was  made,  with  the 
computed  significant  word  underlined;  2)  the  question 
(Q)  resulting  from  the  sentence  analysis;  3)  the  formally 
correct  answer  (A)  to  the  question  (this  answer  is  based  on 
the  generic  response);  and  4)  four  numbers  found  under 
A through  D in  which  A indicates  the  significance  rank- 
ing of  the  underlined  significant  word,  B the  occurrence 
order  of  each  significant  word,  C the  word  number  of  the 
significant  word  within  the  text,  and  D the  constituent 
number  that  was  randomly  selected  to  generate  the  question. 
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PAST  PRES  6 6 genug  Material  fllr  Aufsatz  zurUckgeben 


Der  Franzose  hatte  einige  Stunden  vor  den  Kamelen  im  Zoo  verbracht. 
Wie  lange  hatte  der  Franzose  vor  den  Kamelen  im  Zoo  verbracht? 


kbnnen  PRES  weil  wohnen  6 Amerikaner  in  Boston  besuchen 


Zwischenzeit  hatte  der  Franzose  seine  Arbeit  schon  lHngst  beendet . 


ion  langst  beenden  0 Franzose  0 Arbeit  in  Zwischenzeit 


dlirfen  hereinkoamen  6 niemand  ausser  Frau  von  Deutschen  in  Studierzinuner 


schnell  stellte  der  Amerikaner  mit  Hilfe  seiner  Notizen  eine  Beschreibung  des  Kamels  zusammen. 


schnell  zusammenstellen  0 Amerikaner  0 Beschreibung  von  Kamel  mit  Hilfe  von  Notizen 


schen  0 Essen  zu  Deutschen  koramen 


rauchen 


endlich  koranen  PRES  6 0 Tag  fllr  Fabrikation 


■ Franzose  in  ein  nahes  Cafe  gegangen,  um  die  Skizze  auszuarbeiten. 
Cafe  war  der  Franzose  schnell  gegangen,  um  die  Skizze  auszuarbeiten? 


wollen  gewinnen  0 Kandidaten  0 Preis 


Der  Aaerikaner  ist  Mitglied  von  zehn  Klubs. 
Ker  ist  Mitglied  von  zehn  Klubs? 


PRES  6 0 Amerikaner  6 Mitglied  von  zehn  Klubs 


PAST  packen  0 Englander  PRES  praktisch  0 Englander  PAST  trauen  0 Englander  mehr  als  0 Augen  Gehim  0 Sachen 


■MM 


haben  0 Englander  0 genug  Material  PRES  0 0 Material  fUr  Aufsatz  nach  fUnf  Wochen 
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